1"use strict";(self.webpackChunkrspamd_docs=self.webpackChunkrspamd_docs||[]).push([[8130],{77735:e=>{e.exports=JSON.parse('{"archive":{"blogPosts":[{"id":"2025/10/18/incident-disclosure","metadata":{"permalink":"/blog/2025/10/18/incident-disclosure","editUrl":"https://github.com/rspamd/docs.rspamd.com/edit/master/blog/2025-10-18-incident-disclosure.md","source":"@site/blog/2025-10-18-incident-disclosure.md","title":"Incident Disclosure - Rspamd Public Service Temporary Suspension","description":"On Saturday, October 18th, 2025, the public Rspamd DNSBL RBL feed and what\'s more important public fuzzy service was disrupted due to an unexpected server block by our hosting provider, Hetzner. This interruption affected hundreds of thousands of users and likely led to increased volumes of spam for many legitimate email services worldwide.","date":"2025-10-18T00:00:00.000Z","tags":[{"inline":true,"label":"incident","permalink":"/blog/tags/incident"},{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"},{"inline":true,"label":"service disruption","permalink":"/blog/tags/service-disruption"}],"readingTime":2.13,"hasTruncateMarker":true,"authors":[{"name":"Rspamd Team","title":"Rspamd Maintainers","url":"https://github.com/rspamd","imageURL":"https://github.com/rspamd.png","key":"rspamd","page":null}],"frontMatter":{"slug":"2025/10/18/incident-disclosure","title":"Incident Disclosure - Rspamd Public Service Temporary Suspension","authors":"rspamd","tags":["incident","announcement","service disruption"]},"unlisted":false,"nextItem":{"title":"Rspamd Performance Measures","permalink":"/blog/rspamd-performance"}},"content":"On Saturday, October 18th, 2025, the public Rspamd DNSBL RBL feed and what\'s more important **public fuzzy** service was disrupted due to an unexpected server block by our hosting provider, Hetzner. This interruption affected hundreds of thousands of users and likely led to increased volumes of spam for many legitimate email services worldwide.\\n\\n\x3c!--truncate--\x3e\\n\\n## What Happened\\n\\n- Our monitoring detected service degradation early Saturday, followed by Hetzner\'s notification that our server would be blocked due to \\"suspected port scan attacks.\\"\\n- Rspamd\'s service responds to legitimate external requests on port 11335 as part of its normal operation\u2014the traffic flagged was fully expected and non-malicious.\\n- Despite prior communications clarifying this issue (with Hetzner support involved on three separate occasions), our server was nevertheless blocked following a provider-side false positive.\\n- Although Hetzner\'s support eventually agreed that the block was in error and restored service, we received further claims that network activity from our side (including connections to resolvable, legitimate mail service IPs) was improper, citing \\"unrouted addresses\\" \u2014 a claim contradicted by publicly available DNS records and usage reports.\\n\\n## Why This Matters\\n\\n- This ongoing false positive detection and inflexible incident response directly impacted the reliability of a widely-used open source project, with real-world effects for large-scale email filtering, spam defense, and downstream email service providers.\\n- We have complied with all incident forms, engaged in dialogue multiple times, and transparently reported our findings. Unfortunately, misinterpretation of perfectly ordinary and legitimate network traffic resulted in a block that, while now lifted, has forced us to disable certain public-facing Rspamd services to avoid future risk.\\n\\n## Our Next Steps\\n\\n- We are reviewing our hosting provider relationship and will migrate to a more robust and responsive provider to avoid future interruptions.\\n- Public access to some Rspamd services will remain suspended until a solution that meets both our standards and availability requirements is fully in place.\\n\\n## \u2705 Positive Update\\n\\nDue to the generous support from many companies and individuals who are eager to support the project, we expect the **public fuzzy service to be back online in approximately one week**. We are grateful for the community\'s overwhelming response and commitment to keeping Rspamd services available.\\n\\n## Conclusion\\n\\nThis episode highlights the critical importance of technical nuance and responsive incident handling in the infrastru
1cture supporting public open source projects.\\n\\nWe regret the inconvenience to our user base and assure everyone that maintaining reliable, high-quality service for the global community remains our top priority.\\n\\nThank you for your understanding and support.\\n\\n---\\n\\n*For questions or concerns, please reach out through our [support channels](/support).*"},{"id":"rspamd-performance","metadata":{"permalink":"/blog/rspamd-performance","editUrl":"https://github.com/rspamd/docs.rspamd.com/edit/master/blog/2019-05-16-rspamd-performance.md","source":"@site/blog/2019-05-16-rspamd-performance.md","title":"Rspamd Performance Measures","description":"Preface","date":"2019-05-16T00:00:00.000Z","tags":[{"inline":true,"label":"performance","permalink":"/blog/tags/performance"},{"inline":true,"label":"benchmarks","permalink":"/blog/tags/benchmarks"}],"readingTime":8.04,"hasTruncateMarker":true,"authors":[{"name":"Rspamd Team","title":"Rspamd Maintainers","url":"https://github.com/rspamd","imageURL":"https://github.com/rspamd.png","key":"rspamd","page":null}],"frontMatter":{"slug":"rspamd-performance","title":"Rspamd Performance Measures","authors":["rspamd"],"tags":["performance","benchmarks"]},"unlisted":false,"prevItem":{"title":"Incident Disclosure - Rspamd Public Service Temporary Suspension","permalink":"/blog/2025/10/18/incident-disclosure"}},"content":"## Preface\\n\\nRspamd has always been oriented on the performance but it was always quite hard to measure how fast it was as normally it runs *just fast enough*.\\n\\nHowever, I was recently offered to process [Abusix Intelligence](https://www.abusix.ai/) feeds using Rspamd. These feeds are used to improve Rspamd fuzzy storage quality, to feed URLs and Emails to the DNS black lists provided by Rspamd project and used in SURBL module.\\n\\n\x3c!--truncate--\x3e\\n\\n## Problem statement\\n\\nThe amount of data that required to be processing is huge - it is about 100 millions of messages per day.\\n\\nHere is an example to calculate connections count when processing these messages using Rspamd:\\n\\n<pre>\\n<div class=\\"term\\">\\n$ rspamc stat | \\\\\\n grep \'Connections count\' | \\\\\\n cut -d\' \' -f3 ; \\\\\\n sleep 10 ; \\\\\\n rspamc stat | \\\\\\n grep \'Connections count\' | \\\\\\n cut -d\' \' -f3\\n23548811\\n23564384\\n</div>\\n</pre>\\n\\nIt means that over 10 seconds Rspamd has to process around 15 thousands of messages which gives us a rate of **1500 messages per second**.\\n\\n## Rspamd setup\\n\\nThe settings used to process this amount of messages are pretty similar to those that are provided by default.\\n\\nThere is also some significant amount of home-crafted scripts written in Lua to provide the following functionality:\\n\\n* Provides deduplication to save time on processing of duplicates\\n* Performs conditional checks for url and emails blacklisting:\\n - checks if an url is in whitelists (around 5 whitelists stored in Redis are used)\\n - check if an url is already listed\\n - check if it matches any suspicious patterns\\n* Checks if a message should be learned on fuzzy storage (various conditions)\\n* Stores messages in IMAP folders providing sorting, partitioning and sampling logic\\n* Doing various HTTP and Redis queries for servicing purposes\\n\\n## Hardware\\n\\nNow some words about hardware being used.\\n\\nPreviously we have set the same setup on a small instance of [AX-60](https://web.archive.org/web/20190319181059/https://www.hetzner.com/dedicated-rootserver/ax60-ssd) and it was loaded for around 80%. We have decided to move to a more powerful server to have some margin for processing more emails and doing some experiments.\\n\\nHence, we now have an [AX-160](https://web.archive.org/web/20190319181053/https://www.hetzner.com/dedicated-rootserver/ax160-ssd) AMD server rented in [Hetzner](https://www.hetzner.com/). This is quite a powerful machine and the current load pictures look like this one:\\n\\n<pre>\\n<div class=\\"term\\">\\ntop - 14:36:26 up 23:26, 1 user, load average: 15.76, 13.22, 12.46\\nTasks: 511 total, 3 running, 508 sleeping, 0 stopped, 0 zombie\\n%Cpu(s): 14.1 us, 4.6 sy, 0.0 ni, 78.9 id, 0.0 wa, 0.0 hi, 2.4 si, 0.0 st\\nMiB Mem : 128802.5 total, 56985.7 free, 27897.5 used, 43919.3 buff/cache\\nMiB Swap: 4092.0 total, 3925.5 free, 166.5 used. 100018.6 avail Mem\\n\\n PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\\n 14085 unbound 20 0 2058412 1.6g 6852 S 131.2 1.3 1478:04 unbound\\n 66509 _rspamd 20 0 806976 733336 23592 S 68.8 0.6 169:52.21 rspamd\\n 66498 _rspamd 20 0 780144 699540 23852 S 62.5 0.5 156:19.14 rspamd\\n 66502 _rspamd 20 0 816152 744352 23796 S 56.2 0.6 164:26.39 rspamd\\n 66468 _rspamd 20 0 773532 697084 23736 S 50.0 0.5 117:36.32 rspamd\\n 66491 _rspamd 20 0 806652 722340 23728 S 50.0 0.5 148:04.54 r
1spamd\\n 66476 _rspamd 20 0 767300 705996 23596 S 43.8 0.5 129:04.30 rspamd\\n 66481 _rspamd 20 0 797944 730528 23896 S 43.8 0.6 139:34.35 rspamd\\n 66443 _rspamd 20 0 727632 657104 23372 S 37.5 0.5 88:39.26 rspamd\\n 66451 _rspamd 20 0 742192 665196 23632 S 37.5 0.5 94:49.75 rspamd\\n 66456 _rspamd 20 0 790908 725784 23488 S 37.5 0.6 101:32.06 rspamd\\n 66463 _rspamd 20 0 771540 696064 23692 S 37.5 0.5 108:08.65 rspamd\\n 66487 _rspamd 20 0 780220 713024 23428 S 37.5 0.5 144:51.79 rspamd\\n 66447 _rspamd 20 0 762440 689592 23736 S 31.2 0.5 90:23.93 rspamd\\n 66455 _rspamd 20 0 763520 696108 23580 S 31.2 0.5 97:57.57 rspamd\\n 66464 _rspamd 20 0 764644 688724 23696 S 31.2 0.5 111:32.74 rspamd\\n 66469 _rspamd 20 0 756952 678704 23612 S 31.2 0.5 127:55.02 rspamd\\n127011 rbldns 20 0 358824 307700 2244 R 31.2 0.2 10:26.14 rbldnsd\\n 10767 redis 20 0 9912104 7.7g 2532 S 25.0 6.1 236:29.63 redis-server\\n 66438 _rspamd 20 0 746772 680624 23424 R 25.0 0.5 82:18.04 rspamd\\n 66433 _rspamd 20 0 751180 687244 23472 S 18.8 0.5 80:12.21 rspamd\\n 66437 _rspamd 20 0 737200 669428 23796 S 18.8 0.5 81:37.81 rspamd\\n 10671 stunnel4 20 0 24.0g 77252 3644 S 12.5 0.1 269:06.53 stunnel4\\n 26994 root 20 0 11900 3984 3072 R 12.5 0.0 0:00.02 top\\n 66442 _rspamd 20 0 808808 707020 23608 S 12.5 0.5 85:11.64 rspamd\\n 17821 clickho+ 20 0 21.8g 3.9g 18964 S 6.2 3.1 116:13.04 clickhouse-serv\\n</div>\\n</pre>\\n\\nRspamd is also being fed via proxy worker that runs on another host and performs initial data collection and emitting messages via the Internet providing transport encryption using HTTPCrypt. However, its CPU usage is quite negligible - it uses only a single CPU core by around 40% in average.\\n\\n## Results analytics\\n\\nAs you can see, this machine runs also [Clickhouse](https://clickhouse.yandex), Redis, own recursive resolver (Unbound), and it still has **~80% idle** processing these **1500 messages per second**.\\n\\nIf we look at the performance counters by attaching to some of the worker processes, we would see the following picture:\\n\\n<pre>\\n<div class=\\"term\\">\\n# timeout 30 perf record -p 66481\\n[ perf record: Woken up 1 times to write data ]\\n[ perf record: Captured and wrote 1.171 MB perf.data (29833 samples) ]\\n# perf report\\n\\n# Overhead Command Shared Object Symbol\\n# ........ ....... ....................... .......................................................................\\n#\\n 5.23% rspamd rspamd [.] lj_alloc_free\\n 3.35% rspamd rspamd [.] lj_str_new\\n 3.03% rspamd librspamd-server.so [.] gc_sweep\\n 2.20% rspamd rspamd [.] lj_alloc_malloc\\n 1.94% rspamd rspamd [.] gc_sweep\\n 1.50% rspamd libc-2.28.so [.] __strlen_avx2\\n 1.32% rspamd rspamd [.] release_unused_segments\\n 1.24% rspamd rspamd [.] lj_BC_TGETS\\n 1.17% rspamd libjemalloc.so.2 [.] free\\n 1.04% rspamd librspamd-server.so [.] lj_BC_JLOOP\\n 1.03% rspamd librspamd-server.so [.] propagatemark\\n 1.01% rspamd libpthread-2.28.so [.] __pthread_mutex_lock\\n 1.01% rspamd libglib-2.0.so.0.5800.3 [.] g_hash_table_lookup\\n 0.94% rspamd libjemalloc.so.2 [.] malloc\\n 0.77% rspamd rspamd [.] lj_func_newL_gc\\n 0.76% rspamd rspamd [.] propagatemark\\n 0.75% rspamd rspamd [.] lj_tab_get\\n 0.69% rspamd libpthread-2.28.so [.] __pthread_mutex_unlock_usercnt\\n 0.65% rspamd librspamd-server.so [.] t1ha2_atonce\\n 0.61% rspamd librspamd-server.so [.] newtab\\n 0.60% rspamd libicui18n.so.63.1 [.] icu_63::NGramParser::search\\n 0.59% rspamd [kernel.kallsym
1s] [k] copy_user_generic_string\\n 0.58% rspamd librspamd-server.so [.] match\\n 0.58% rspamd librspamd-server.so [.] lj_tab_new1\\n 0.56% rspamd librspamd-server.so [.] rspamd_task_find_symbol_result\\n 0.52% rspamd [kernel.kallsyms] [k] _raw_spin_lock_irqsave\\n 0.48% rspamd librspamd-server.so [.] rspamd_vprintf_common\\n 0.46% rspamd librspamd-server.so [.] lj_str_new\\n 0.42% rspamd rspamd [.] index2adr\\n 0.42% rspamd rspamd [.] lj_BC_CALL\\n 0.42% rspamd libc-2.28.so [.] __strcmp_avx2\\n 0.42% rspamd libc-2.28.so [.] __memmove_avx_unaligned_erms\\n</div>\\n</pre>\\n\\nThe top consumers are Lua allocator and garbage collector. Since we are using [Rspamd experimental package]({{ site.baseurl }}/downloads.html) on Debian Buster, then it is built with bundled [LuaJIT 2.1 beta3](https://luajit.org) and Jemalloc allocator, however, it seems that there is some issue with this allocator in Debian Buster, so I had to load it manually via the following command:\\n\\n<pre>\\n<div class=\\"term\\">\\n# systemctl edit rspamd.service\\n\\n[Service]\\nEnvironment=\\"LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2\\"\\n</div>\\n</pre>\\n\\nFollowed by restarting of Rspamd.\\n\\nIt is interesting that this Rspamd setup accepts all connections encrypted using [HTTPCrypt](https://highsecure.ru/httpcrypt.pdf) but `chacha_blocks_avx2` takes less than 0.16% of CPU according to `perf` report.\\n\\nThis particular instance of Rspamd is slightly tuned to use more memory to save some CPU cycles:\\n\\n~~~hcl\\n# local.d/options.inc\\n\\nlua_gc_step = 100;\\nlua_gc_pause = 400;\\nfull_gc_iters = 10000;\\n~~~\\n\\nThese options tell Rspamd to preserve Lua objects in memory for longer time, at the same time in this mode, we can also observe GC stats on workers that performs full GC loop each 10k messages being scanned:\\n\\n<pre>\\n<div class=\\"term\\">\\n$ tail -f /var/log/rspamd/rspamd.log | fgrep \'full gc\'\\n\\nperform full gc cycle; memory stats: 58.66MiB allocated, 62.01MiB active, 6.08MiB metadata, 84.71MiB resident, 90.64MiB mapped; lua memory: 107377 kb -> 38015 kb; 308.0022420035675 ms for gc iter\\n</div>\\n</pre>\\n\\nAs you can see, full GC iter takes quite a significant time. However, it still keeps Lua memory usage sane. The ideas behind this GC mode have been taken from the generational GC idea in [LuaJIT Wiki](http://wiki.luajit.org/New-Garbage-Collector#gc-algorithms_generational-gc).\\n\\n## Resulting graphs\\n\\nHere are some UI captures taken from a previous machine:\\n\\n<img width=\\"75%\\" class=\\"img-fluid\\" src=\\"/img/perf_webui1.png\\">\\n\\n<img width=\\"75%\\" class=\\"img-fluid\\" src=\\"/img/perf_webui2.png\\">\\n\\nAs you can observe, there was some HAM portion increase over the recent days, however, it was caused by adding new sampling logic and duplicates filtering to save CPU resources (these messages are marked as ham and excepted from scan).\\n\\nThere is also a [Clickhouse](https://clickhouse.yandex) based dashboard that\'s created using [Redash](https://redash.io):\\n\\n<img width=\\"75%\\" class=\\"img-fluid\\" src=\\"/img/perf_redash.png\\">\\n\\nSince we have Clickhouse on board, we can do various analytics. Here is an average scan time for messages:\\n\\n<pre>\\n<div class=\\"term\\">\\n:) select avg(ScanTimeVirtual) from rspamd where Date=today();\\n\\nSELECT avg(ScanTimeVirtual)\\nFROM rspamd\\nWHERE Date = today()\\n\\n\u250c\u2500avg(ScanTimeVirtual)\u2500\u2510\\n\u2502 95.62269064131341 \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n</div>\\n</pre>\\n\\n... and average size of messages:\\n\\n\\n<pre>\\n<div class=\\"term\\">\\n:) select median(ScanTimeVirtual) from rspamd where Date=today();\\n\\n:) select avg(Size) from rspamd where Date=today();\\n\\nSELECT avg(Size)\\nFROM rspamd\\nWHERE Date = today()\\n\\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500avg(Size)\u2500\u2510\\n\u2502 1778.31 \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n</div>\\n</pre>\\n\\n## Conclusions\\n\\nSo with this load rate (**1500 messages per second**) and with the average size of messages around 2Kb, Rspamd processes each message in around **100ms in average**. I hope these numbers could give one some impression about Rspamd performance in general.\\n\\nI would like to give the main kudos to [Abusix](https://www.abusix.com) who are constantly supporting Rspamd project and have generously provided their amazing spam feeds to improve Rspamd quality!"}]}}')}}]);
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.