1"use strict";(globalThis.webpackChunkpublic_docs=globalThis.webpackChunkpublic_docs||[]).push([[1038],{9296(e){e.exports=JSON.parse('{"archive":{"blogPosts":[{"id":"panda","metadata":{"permalink":"/posts/panda","source":"@site/blog/2026-06-03-panda/index.md","title":"Panda: one tool for the whole data stack","description":"How we stopped drowning our agents in JSON.","date":"2026-06-03T00:00:00.000Z","tags":[{"inline":true,"label":"panda","permalink":"/posts/tags/panda"},{"inline":true,"label":"ai","permalink":"/posts/tags/ai"},{"inline":true,"label":"mcp","permalink":"/posts/tags/mcp"},{"inline":true,"label":"data","permalink":"/posts/tags/data"},{"inline":true,"label":"clickhouse","permalink":"/posts/tags/clickhouse"},{"inline":true,"label":"llm","permalink":"/posts/tags/llm"}],"readingTime":9.74,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"qu0b","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/qu0b","github":"https://github.com/qu0b","twitter":"https://x.com/stefan_star","imageURL":"/img/team/qu0b.jpeg","key":"qu0b","page":null}],"frontMatter":{"slug":"panda","title":"Panda: one tool for the whole data stack","authors":["samcm","qu0b"],"description":"How we stopped drowning our agents in JSON.","tags":["panda","ai","mcp","data","clickhouse","llm"],"image":"/img/blog/panda/header.png","githubRepos":[{"name":"panda","url":"https://github.com/ethpandaops/panda"}]},"unlisted":false,"nextItem":{"title":"Replaying a year of mainnet with the Fast Confirmation Rule","permalink":"/posts/fcr-simulator"}},"content":"## Intro\\n\\nOver the last decade, the way engineers interacted with the observability data of the platform they were building converged to a similar point: a handful of datasources wired into a central visualization tool, usually Grafana. When it came time to find an answer, sometimes a dashboard already existed that solved your exact question. Other times you\'d be writing a one-off query and praying you hadn\'t overlooked something.\\n\\nQuerying the ethPandaOps stack has looked very similar since our team was conceived in 2022. Prometheus metrics lived in Victoriametrics, logs in Loki, Xatu data lived in ClickHouse, all accessible via Grafana. Each had a different query language and correlating data between sources was painful and tedious. This process also involved a human in the loop by relying on an authenticated web browser to click through the Grafana UI resulting in friction for agents.\\n\\nWe built [Panda](https://github.com/ethpandaops/panda) to solve these problems in an opinionated and streamlined way with the goal of unlocking new ways of working. Panda collapses everything into a single interface. It\'s a CLI, an MCP server, and a sandboxed Python runtime that already knows how to reach every datasource we run, and through our hosted credential proxy it\'s open to any Ethereum core dev. This post walks through how it works, and how it became part of our daily workflow.\\n\\n## What didn\'t work\\n\\nThe obvious way to wire a data stack into an LLM is one MCP tool per datasource. We built exactly that a year ago: an [ethpandaops-data](https://github.com/ethpandaops/ai-cookbook/tree/master/tools/mcp-servers/ethpandaops-data) MCP server with a `loki_tool`, a `prometheus_tool`, and a `clickhouse_tool`, each pointed at our Grafana datasources.\\n\\nIt sort of worked, but it had a massive problem. Every tool would hand its raw result straight back to the model, serialized as pretty-printed JSON. Ask the `clickhouse_tool` for blob counts per block over the last day and ~7000 rows of JSON would land straight in the context window. Not only was this extremely expensive, but it was also limiting the complexity and reliability of our agents, as context rot would corrupt the original question and the agent would get confused (sometimes completely forgetting its original task after firing off a single query!)\\n\\nThis was also limited in its usefulness. An agent with ~7000 rows in its context couldn\'t easily calculate an aggregate over all the rows. We experimented with ideas like piping through `jq`, but found it was too rigid for all our use-cases.\\n\\n## Panda: Empowering Agents\\n\\nPanda gives the model a runtime environment instead of a pile of tools. The MCP surface is just three tools, each mirrored by a CLI command:\\n\\n- `execute_python` (`panda execute`) runs code in a sandbox\\n- `manage_session` (`panda session`) keeps a
1sandbox warm between calls\\n- `search` (`panda search`) does semantic search over query examples, runbooks, EIPs, and consensus specs\\n\\nEverything past that is a Python library inside the sandbox, and it spans the whole stack: ClickHouse, Prometheus, Loki, Dora, and the beacon and execution nodes. The blob query that flooded the window above comes back as a summary instead:\\n\\n```bash\\npanda execute --code \'\\nfrom ethpandaops import clickhouse\\n\\ndf = clickhouse.query(\\"xatu-cbt\\", \\"\\"\\"\\n SELECT blob_count\\n FROM mainnet.fct_block_blob_count_head FINAL\\n WHERE slot_start_date_time >= now() - INTERVAL 1 DAY\\n\\"\\"\\")\\nprint(df[\\"blob_count\\"].describe())\\n\'\\n```\\n\\nThe ~7000 rows live and die inside the sandbox, and only the eight-line summary that `describe()` prints ever travels back to the model. When the model needs a loop or a join, that runs in the sandbox too, instead of turning into hundreds of tool calls with every intermediate row passing back through the window.\\n\\nAnthropic wrote this pattern up well in [Code execution with MCP](https://www.anthropic.com/engineering/code-execution-with-mcp), and **we strongly recommend it.** Today\'s models are very good at writing code so this was an overnight productivity multiplier for us.\\n\\n## Architecture\\n\\nOur observability platform has a unique access pattern: we aren\'t the only ones who use it! Many Ethereum core devs use our data daily to check in on how the 854th iteration of their devnet is performing, or if Mainnet is still kicking. Handing out individual credentials to all of them isn\'t maintainable, and not all of the tools we use allow for fine-grained permissions.\\n\\nWe were also now giving agents direct access to our backend infrastructure like VictoriaMetrics, Clickhouse and Loki. This meant that an agent could trivally pull these credentials from its runtime environment, leaking them to the inference provider.\\n\\nTo solve these issues we split Panda in to multiple components:\\n\\n<img src=\\"/img/blog/panda/architecture.png\\" alt=\\"Request flow: a sandbox and server on your machine send a request to a credential-holding proxy, which queries ClickHouse, Prometheus, Loki, and an Eth node and returns rows. The sandbox holds no credentials.\\" />\\n\\n### Server\\nServer is the main interaction point for both humans and agents. It exposes an MCP & CLI interface that provides access to the rest of the components. It\'s responsible for negotiating auth against the `proxy`. It runs on your machine and is the glue that brings everything else together.\\n\\n### Credential Proxy\\nThe proxy is the only thing that ever holds a password for our datasources. Our team hosts an instance of `proxy`, which is what actually talks to our backend services like Clickhouse, Victoriametrics and Loki. It validates authentication tokens, and only shows the user the datasources the user actually has access to. It also gives us a consistent observability surface for all these datasources so we can see what\'s actually being used.\\n\\nFor Panda users, the onboarding flow looks something like this: you sign in to the hosted proxy with GitHub, your queries run locally on your own machine, and you don\'t have to worry about any credentials. New datasources will appear in Panda automatically as our team adds them.\\n\\n### Sandbox\\n\\nThe sandbox does the grunt work. It\'s a Python container that runs on your own machine as we\'ve found that compute best kept close to whoever is asking the question. It uses a custom docker image that is shipped alongside Panda, which includes Python wrappers for interacting with our datasources.\\n\\nThe code running in the sandbox was probably written by an LLM and it\'s about to be executed, so the sandbox treats it as hostile. Every container runs:\\n\\n- as `nobody`, with a read-only root filesystem\\n- with every Linux capability dropped and `no-new-privileges` set\\n- under pid, memory, and CPU limits\\n- on an isolated Docker network rather than the host network\\n\\nThe sandbox holds no credentials in the first place. It gets an API URL and a token scoped to the local server and nothing more, so even if the generated code went looking for a secret to exfiltrate, there is none in the container to take. The writable space is a small: `/tmp`, `/workspace` that survives between calls in the same session, and an `/output` mou
1nt for the files you want to pull back out.\\n\\n##### Sessions\\n\\nA sandbox isn\'t thrown away after a single call. When passing the `session_id` from the previous response into the next call, you get the same warm container back, with your imports still loaded and anything you wrote to `/workspace` still on disk. A sequence of `execute_python` calls becomes one analysis that can build on itself.\\n\\nThis pays off when one step is expensive and the next few want to reuse the result. The first call caches 30 days of block proposers to a parquet file:\\n\\n```bash\\npanda execute --code \'\\nfrom ethpandaops import clickhouse\\n\\nproposers = clickhouse.query(\\"xatu-cbt\\", \\"\\"\\"\\n SELECT slot, proposer_validator_index AS validator\\n FROM mainnet.fct_block_proposer FINAL\\n WHERE slot_start_date_time >= now() - INTERVAL 30 DAY\\n\\"\\"\\")\\nproposers.to_parquet(\\"/workspace/proposers.parquet\\")\\nprint(len(proposers), \\"proposals cached\\")\\n\'\\n```\\n\\nA later call in the same session reads that file straight back and joins it against a second query, without paying for the first one again:\\n\\n```bash\\npanda execute --session 9bdd56e2 --code \'\\nimport pandas as pd\\nfrom ethpandaops import clickhouse\\n\\nproposers = pd.read_parquet(\\"/workspace/proposers.parquet\\") # still here from the last call\\ncorrectness = clickhouse.query(\\"xatu-cbt\\", \\"\\"\\"\\n SELECT attesting_validator_index AS validator,\\n avg(inclusion_distance) AS avg_inclusion_distance\\n FROM mainnet.fct_attestation_correctness_by_validator_canonical FINAL\\n WHERE slot_start_date_time >= now() - INTERVAL 30 DAY\\n GROUP BY validator\\n\\"\\"\\")\\njoined = proposers.merge(correctness, on=\\"validator\\", how=\\"left\\")\\nprint(joined[\\"avg_inclusion_distance\\"].describe())\\n\'\\n```\\n\\n## Discovery\\n\\nUnfortunately we still had a discovery problem. Agents (and humans!) are non deterministic. We needed a way to guide Panda users to the correct workflows without front-loading their context.\\n\\nTo begin with, we created `panda getting-started` which acts as the first-hop for the user to learn how to get more information. All datasources are able to expose documentation around the data they provide. For instance, the Clickhouse datasource exposes schemas of the tables it\'s connected to via `panda schema $table`.\\n\\nThis covered what exists, but exact workflows are more complex. We landed on a document embedding index of 4 different types, allowing us to run semantic search across them.\\n\\n- query examples (individual SQL, PromQL, and LogQL snippets)\\n- investigation runbooks (multi step procedures)\\n- EIPs\\n- consensus specs (protocol constants and spec text).\\n\\nThis is what lets an agent resolve a vague ask like \\"get distinct validators per slot from aggregate_and_proof\\" reliably, without knowing the exact table names up front. It also gives our team one place to capture the patterns and runbooks we\'ve built up over the years, ready for either of us to reuse.\\n\\n```bash\\npanda search runbooks \\"network not finalizing\\"\\npanda search eips \\"blob base fee\\"\\npanda search consensus-specs \\"MAX_EFFECTIVE_BALANCE\\"\\npanda search queries \\"block arrival\\"\\n```\\n\\nThe consensus specs go one step further and are a direct call away inside the sandbox, so a query can resolve a constant inline instead of hard-coding it:\\n\\n```bash\\npanda execute --code \'\\nfrom ethpandaops import specs\\nprint(specs.get_constant(\\"MAX_EFFECTIVE_BALANCE\\"))\\nprint(specs.get_spec(\\"deneb\\", \\"beacon-chain\\")[\\"title\\"])\\n\'\\n```\\n\\n## CLI or MCP?\\n\\n**We don\'t care either way.**\\n\\nThe same server sits behind the MCP tools and the Panda CLI, so they share a library, a proxy, and the credential boundary. An agent can call the MCP tools directly or shell out to the CLI, the same as a human would.\\n\\nThe CLI gives you two ways to reach the data: `panda clickhouse query` proxies a single raw query straight to the datasource, while `panda execute` runs your code in the sandbox first. Frontier models are also smart enough to choose on the fly.\\n\\n```bash\\npanda clickhouse query xatu-cbt \\"SELECT count() FROM mainnet.fct_block FINAL\\"\\npanda execute --code \'from ethpandaops import clickhouse; print(clickhouse.query(\\"xatu-cbt\\", \\"SELECT count() FROM mainnet.fct_block FINAL\\"))\'\\n```\\n\\n## How we use it\\n\\nIn practice, Panda is a tool we reach for every day when dealing with both the Ethereum protocol and our internal platform. Usually these are one-off questions that were previously too tedious to investigate: \\"How many attestations arrive after the aggregation deadline if the aggregator was in Australia compared to Europe?\\" We write some of the longer ones up at [investigations.ethpandaops.io](https://investigations.ethpandaops.io).\\n\\nA recent query for our internal platform was \\"Why is kafka lag increasing?\\". The agent used Panda to check Victoriametrics to find why topics were lagging via Prometheus metrics, moved on to a Loki query for error logs from the consumer instances, and finally concluded that ClickHouse\'s write throughput was saturated via Clickhouse\'s metrics.\\n\\n**We are increasingly using agents to augment our use of Panda.** It\'s usually much easier to ask Claude to interact with Panda than to run commands with bash ourselves.\\n\\n## Check it out\\n\\nFor those that are curious as to how Panda works under the hood: the full source, the self-hostable proxy, and the architecture docs all live at [github.com/ethpandaops/panda](https://github.com/ethpandaops/panda). If you\'re an Ethereum Core Dev, you can use the ethPandaOps proxy already via `panda init`.\\n\\n------\\n\\nLove,\\n\\nethPandaOps Team"},{"id":"fcr-simulator","metadata":{"permalink":"/posts/fcr-simulator","source":"@site/blog/2026-05-26-fcr-simulator/index.md","title":"Replaying a year of mainnet with the Fast Confirmation Rule","description":"We replayed 12 months of mainnet beacon chain history through 4 consensus-layer implementations of the Fast Confirmation Rule.","date":"2026-05-26T00:00:00.000Z","tags":[{"inline":true,"label":"fcr","permalink":"/posts/tags/fcr"},{"inline":true,"label":"consensus-layer","permalink":"/posts/tags/consensus-layer"},{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"kubernetes","permalink":"/posts/tags/kubernetes"},{"inline":true,"label":"data","permalink":"/posts/tags/data"}],"readingTime":5.13,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","t
1witter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null}],"frontMatter":{"slug":"fcr-simulator","title":"Replaying a year of mainnet with the Fast Confirmation Rule","authors":["samcm","savid"],"description":"We replayed 12 months of mainnet beacon chain history through 4 consensus-layer implementations of the Fast Confirmation Rule.","tags":["fcr","consensus-layer","xatu","kubernetes","data"],"image":"/img/blog/fcr-simulator/header.png","githubRepos":[{"name":"fcr-simulator","url":"https://github.com/ethpandaops/fcr-simulator"}]},"unlisted":false,"prevItem":{"title":"Panda: one tool for the whole data stack","permalink":"/posts/panda"},"nextItem":{"title":"Validator Report: Investigate Validator Performance","permalink":"/posts/validator-report"}},"content":"import MonthlyFcrRateChart from \'@site/src/components/blog/fcr-simulator/MonthlyFcrRateChart\';\\nimport ConfirmDistanceCDF from \'@site/src/components/blog/fcr-simulator/ConfirmDistanceCDF\';\\nimport ConfirmDistanceTimeSeries from \'@site/src/components/blog/fcr-simulator/ConfirmDistanceTimeSeries\';\\n\\nWe built [fcr-simulator](https://github.com/ethpandaops/fcr-simulator) to answer a simple question: if the [Fast Confirmation Rule](https://fastconfirm.it) had already been running on mainnet, what would it have done?\\n\\nTo study this, we replayed historical mainnet beacon chain data through 4 consensus-layer client implementations of FCR. Each client saw the same blocks, the same attestations, and the same cutoff rules. We then compared how each implementation classified each slot.\\n\\nThis post describes the simulator, the dataset, the infrastructure we used to run the replays, and the results from the v0.2.0 dataset.\\n\\nFor background on FCR itself, [fastconfirm.it](https://fastconfirm.it) is the best place to start.\\n\\n## The simulator\\n\\nThe simulator is intentionally narrow. Its job is to feed the same mainnet history and attestation data into multiple FCR implementations, then record the result for every slot.\\n\\nAt the centre of the system is an `orchestrator`. The orchestrator takes a job specification, starts an engine such as `lighthouse`, and provides that engine with the data it needs. The engine calls back to the orchestrator over HTTP.\\n\\nThis separation is a large step for reproducibility. The orchestrator is responsible for fetching blocks and attestations, while each client implementation is responsible for applying its FCR logic. That keeps the multi-client simulator easier to reason about, and it means we can swap or replay datasources without reimplementing data loading separately in every client.\\n\\nWe also \\"warm\\" the beacon nodes before the simulation run to hydrate their fork choice store and any caches. For the run below, we warmed each beacon node with 10 epochs worth of attestations. \\n\\n<img src=\\"/img/blog/fcr-simulator/diagram-slot-loop.png\\" alt=\\"Diagram: orchestrator slot loop driving an engine subprocess\\" />\\n\\n## The data\\n\\nThe simulator uses data we already had from other ethpandaops infrastructure:\\n\\n- **Blocks**\\n - Canonical blocks from `era` files, using the Nimbus team\u2019s mirror\\n - Orphaned blocks from [ethpandaops/tracoor](https://tracoor.mainnet.ethpandaops.io)\\n- **Attestations**\\n - Single Attestations from [ethpandaops/xatu](https://github.com/ethpandaops/xatu) that voted for each block before the 12 second cutoff\\n\\n**The attestation dataset is not a perfect recreation of what a beacon node would have experienced at the time**. For some slots, this view of the network may report more attestations than your average beacon node. Conversely, for other slots, this view may have less attestations. This data also completely ignores *aggregate* attestations. We assume that attestation aggregation is steady and reliable on Mainnet as this is where most nodes would get 62/64 of their attestations from.\\n\\n## Running the simulations\\n\\nReplaying a year of mainnet through fork choice one slot at a time would be too slow to be useful. The slot state transition is expensive, and repeated queries against the Xatu database would add more overhead.\\n\\nTo make the experiment practical, we ran the replay as a sharded Kubernetes job. We also ran a one-off query against X
1atu and stored the result in our in-house S3 cluster. This gave every run the same input data and avoided repeatedly querying the database during simulation.\\n\\n<img src=\\"/img/blog/fcr-simulator/diagram-sharded-replay.png\\" alt=\\"Diagram: sharded replay topology across the k3s cluster\\" />\\n\\nWe ran the simulations in 1 to 3 month epoch ranges. This was mostly a practical choice: if a job failed near the end of a long run, we wanted to limit how much work could be lost or need to be discarded.\\n\\nAfter benchmarking, we ran 96 beacon nodes across 6 [Hetzner AX162](https://www.hetzner.com/dedicated-rootserver/ax162-r/configurator/) servers. Each server ran 16 beacon nodes. We also mounted each client database into RAMFS to minimize the expensive disk reads and writes.\\n\\n**The workload was heavily memory bound, but this setup still gave us a roughly 25x to 35x speedup compared with a sequential replay.**\\n\\n## Results\\n\\n:::warning[Caveat]\\nAbout 31 hours of slots, or 0.36% of the dataset, are excluded from every chart and number below. This consists of 9 Xatu pipeline issues totalling about 22 hours, and one 8.4 hour simulator feeder bug on 2026-04-01.\\n:::\\n\\n### Coverage\\n\\n| Client | Epoch range | Dates (UTC) | Slots |\\n|------------|---------------------|--------------------------|-----------|\\n| Lighthouse | 369,000 to 449,000 | 2025-05-29 to 2026-05-20 | 2,560,000 |\\n| Lodestar | 417,500 to 449,000 | 2025-12-31 to 2026-05-20 | 1,008,000 |\\n| Grandine | 423,800 to 449,000 | 2026-01-28 to 2026-05-20 | 806,400 |\\n| Teku | 423,800 to 449,000 | 2026-01-28 to 2026-05-20 | 806,400 |\\n\\n<MonthlyFcrRateChart />\\n\\n<ConfirmDistanceCDF />\\n\\n<ConfirmDistanceTimeSeries />\\n\\n### What we found\\n\\nThe most important number first: **zero false confirmations**. Across 5.15 million slot evaluations and all four clients, no implementation ever fast-confirmed a block that turned out to be non-canonical on the finalized chain. The simulator checks each engine\'s `confirmed_root` against the ERA-file canonical block at that slot during the merge step, and across the entire 12-month period (including the December 4 2025 Fusaka mainnet instability) there are zero mismatches.\\n\\nHeadline rate: across 800,000 mainnet slots in the comparable window, roughly **96 out of every 100 slots would have been fast-confirmed within 12 seconds**.\\n\\nThree of the four clients agree almost exactly on which slots qualify. Lighthouse and Grandine report identical 1-slot rates to four decimal places (**95.8165%**) and agree on every single slot in the shared window. Lodestar lands at **95.8164%**, which is a single-slot difference across ~800k. Effectively identical.\\n\\n**Teku is the odd one out.** It reports **97.3926%**, marking ~12,500 *more* slots as fast-confirmed than the others over the same window.\\n\\n## Reproducibility\\n\\nEvery CSV in the v0.2.0 release is pinned to a specific engine image tag. Each engine image tag is pinned to an upstream git ref.\\n\\nThe full mapping is in the release `METADATA.json`, including:\\n\\n- schema\\n- run parameters\\n- attestation source mode\\n- per-CSV image SHA\\n- per-engine upstream commit\\n\\nYou can download the dataset with:\\n\\n```bash\\ncurl -L https://github.com/ethpandaops/fcr-simulator/releases/download/v0.2.0/fcr-data-2026-05-26.tar -o fcr-data.tar\\ntar -xf fcr-data.tar\\n```\\n\\nThe orchestrator and engines are available at https://github.com/ethpandaops/fcr-simulator\\n\\nLove,\\n\\nEthPandaOps Team"},{"id":"validator-report","metadata":{"permalink":"/posts/validator-report","source":"@site/blog/2026-05-21-validator-report/index.md","title":"Validator Report: Investigate Validator Performance","description":"Use the Validator Report page in The Lab to scan validators, catch missed proposals, drill from day to hour to slot, and export Markdown evidence.","date":"2026-05-21T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"validators","permalink":"/posts/tags/validators"},{"inline":true,"label":"lab","permalink":"/posts/tags/lab"},{"inline":true,"label":"consensus-layer","permalink":"/posts/tags/consensus-layer"},{"inline":true,"label":"data","permalink":"/posts/tags/data"}],"readingTime":4.54,"hasTruncateMarker":false,"authors":[{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null}],"frontMatter":{"slug":"validator-report","title":"Validator Report: Investigate Validator Performance","authors":["savid"],"description":"Use the Validator Report page in The Lab to scan validators, catch missed proposals, drill from day to hour to slot, and export Markdown evidence.","tags":["xatu","validators","lab","consensus-layer","data"],"image":"/img/blog/validator-report/header.png","relatedLinks":[{"name":"Example Validator Report","url":"https://lab.ethpandaops.io/ethereum/validators/report?indices=%22768711%2C1915257%2C1670363%2C1299575%2C1362706%2C810474%2C1581037%2C1232935%2C16448
105%2C1730988%2C1329583%2C2061162%22&from=1776729600&to=1779408000"},{"name":"Validator Report","url":"https://lab.ethpandaops.io/ethereum/validators/report"}],"githubRepos":[{"name":"lab","url":"https://github.com/ethpandaops/lab"},{"name":"xatu-cbt","url":"https://github.com/ethpandaops/xatu-cbt"}]},"unlisted":false,"prevItem":{"title":"Replaying a year of mainnet with the Fast Confirmation Rule","permalink":"/posts/fcr-simulator"},"nextItem":{"title":"Introducing EthIQ","permalink":"/posts/introducing-ethiq"}},"content":"Most validator issues start with an alert or an odd looking chart, but it can be difficult to dig in to what\'s actually happening. Usually all you have is a validator index and a rough time range, but digging through the data can be a little overwhelming. That\'s where the new [Validator Report](https://lab.ethpandaops.io/ethereum/validators/report) page in The Lab comes in, allowing you to turn a \\"something looks wrong here\\" report into a specific validator, date, hour, and slot.\\n\\nFor this walkthrough I loaded twelve public mainnet validators into a recent report window. Most of the group looks healthy, but two rows matter once you look closer. This is a common pattern for node operators where data is lost in layers of aggregation.\\n\\n<div style={{display: \'flex\', flexWrap: \'wrap\', gap: \'12px\', justifyContent: \'center\', margin: \'24px 0 32px\'}}>\\n <a href=\\"https://lab.ethpandaops.io/ethereum/validators/report?indices=%22768711%2C1915257%2C1670363%2C1299575%2C1362706%2C810474%2C1581037%2C1232935%2C1644805%2C1730988%2C1329583%2C2061162%22&from=1776729600&to=1779408000\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\" style={{display: \'inline-flex\', alignItems: \'center\', justifyContent: \'center\', padding: \'12px 20px\', backgroundColor: \'var(--ifm-color-primary)\', color: \'white\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: 700}}>Open This Example</a>
1\\n <a href=\\"https://lab.ethpandaops.io/ethereum/validators/report\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\" style={{display: \'inline-flex\', alignItems: \'center\', justifyContent: \'center\', padding: \'12px 20px\', border: \'1px solid var(--ifm-color-primary)\', color: \'var(--ifm-color-primary)\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: 700}}>Open Validator Report</a>\\n</div>\\n\\n<ClickableImage\\n src=\\"/img/blog/validator-report/matrix-clickdown.gif\\"\\n alt=\\"Animated Validator Report matrix drilldown from day to hour to slot rows\\"\\n caption=\\"Click a day cell, then an hour cell; the matrix drills down while the table expands underneath.\\"\\n/>\\n\\n## Find the Outlier\\n\\nThe summary cards at the top gives you a quick glance at the state of your validators.\\n\\n<ClickableImage\\n src=\\"/img/blog/validator-report/outlier-cards.png\\"\\n alt=\\"Validator Report summary cards showing attestation inclusion, vote correctness, sync committee, proposals, and balance\\"\\n caption=\\"The summary is mostly healthy, but the proposal card and head-vote sub-metric break pattern. Screenshots use stricter thresholds; the data stays the same.\\"\\n/>\\n\\nAcross these twelve validators the baseline is solid: 99.94% attestation inclusion, 50 missed duties, 99.74% vote correctness, and every validator active. The soft spot is head votes at 99.3%.\\n\\nThe proposal card shows that we\'ve missed 1 block proposal (7/8). Block proposals are more straight-forward to track, so let\'s start with the missed proposal, then inspect the head-vote thread later.\\n\\n## Catch Proposal Misses\\n\\nValidator `1329583` is the missed-proposal row: 0 / 1. Validator `2061162` is the clean comparison at 2 / 2.\\n\\n<ClickableImage\\n src=\\"/img/blog/validator-report/proposals.png\\"\\n alt=\\"Validator Report table showing missed and successful block proposal counts\\"\\n caption=\\"Validator 1329583 shows 0 / 1 proposals while validator 2061162 shows 2 / 2.\\"\\n/>\\n\\n## Find the Head-Vote Drop\\n\\nWe first look at the per-day performance, and we don\'t exactly find our smoking gun in the Inclusion Delay metric. Swapping over to `Head`, and the overall performance still looks pretty uneventful. We can notice a slight dip for validator `1299575` with May 13 landing at 94% head accuracry. All of the cells here are clickable, but hover it first to get a quick summary.\\n\\n<ClickableImage\\n src=\\"/img/blog/validator-report/matrix-hover.png\\"\\n alt=\\"Validator Report matrix hover tooltip for validator 1299575 head votes on May 13\\"\\n caption=\\"Hover the daily cell first: validator 1299575, Wednesday May 13, 94.0% head votes.\\"\\n/>\\n\\nThe chart flips from days into hours for that exact date, and we finally start to see what\'s happening here.\\n\\n<ClickableImage\\n src=\\"/img/blog/validator-report/matrix-drilldown.png\\"\\n alt=\\"Validator Report matrix clicked into hourly view for May 13\\"\\n caption=\\"One click opens the hourly matrix; the 20:00 UTC tile drops to 44.44% head votes.\\"\\n/>\\n\\nThen open `20:00`. That hour drops to 44.44% head vote correctness with 1.33 average inclusion delay. The slot rows underneath show the actual receipts: missed head votes and one attestation that took `3` slots to land.\\n\\n<ClickableImage\\n src=\\"/img/blog/validator-report/drilldown.png\\"\\n alt=\\"Validator Report drilldown showing validator, day, hour, and slot rows\\"\\n caption=\\"The static receipt: validator 1299575 -> 2026-05-13 -> 20:00 -> slot rows.\\"\\n/>\\n\\nWe\'ve finally found some slots where things went wrong. From here, it\'s time to look at node logs and figure out what happened. For these slots, I\'ve found that it\'s useful to correlate with other data on The Lab, particularly the Slot View.\\n\\n## Save the Trail\\n\\nThe URL is shareable, meaning you can send it to another operator and they\'ll land on the exact same view (down to the slot.) We\'ve also baked in a **Download Report** function, which allows you to export Markdown of the current thresholds. \\n\\n<div style={{display: \'flex\', flexWrap: \'wrap\', gap: \'12px\', justifyContent: \'center\', margin: \'24px 0 32px\'}}>
1\\n <a href=\\"https://lab.ethpandaops.io/ethereum/validators/report?indices=%22768711%2C1915257%2C1670363%2C1299575%2C1362706%2C810474%2C1581037%2C1232935%2C1644805%2C1730988%2C1329583%2C2061162%22&from=1776729600&to=1779408000\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\" style={{display: \'inline-flex\', alignItems: \'center\', justifyContent: \'center\', padding: \'12px 20px\', backgroundColor: \'var(--ifm-color-primary)\', color: \'white\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: 700}}>Open This Example</a>\\n <a href=\\"https://lab.ethpandaops.io/ethereum/validators/report\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\" style={{display: \'inline-flex\', alignItems: \'center\', justifyContent: \'center\', padding: \'12px 20px\', border: \'1px solid var(--ifm-color-primary)\', color: \'var(--ifm-color-primary)\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: 700}}>Open Validator Report</a>\\n</div>\\n\\nLove,\\n\\nEthPandaOps Team"},{"id":"introducing-ethiq","metadata":{"permalink":"/posts/introducing-ethiq","source":"@site/blog/2026-03-05-introducing-ethiq/index.md","title":"Introducing EthIQ","description":"An Ethereum protocol knowledge benchmark for AI models","date":"2026-03-05T00:00:00.000Z","tags":[{"inline":true,"label":"ethiq","permalink":"/posts/tags/ethiq"},{"inline":true,"label":"ai","permalink":"/posts/tags/ai"},{"inline":true,"label":"ethereum","permalink":"/posts/tags/ethereum"},{"inline":true,"label":"benchmark","permalink":"/posts/tags/benchmark"}],"readingTime":5.04,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null},{"name":"qu0b","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/qu0b","github":"https://github.com/qu0b","twitter":"https://x.com/stefan_star","imageURL":"/img/team/qu0b.jpeg","key":"qu0b","page":null}],"frontMatter":{"slug":"introducing-ethiq","title":"Introducing EthIQ","authors":["samcm","parithosh","qu0b"],"description":"An Ethereum protocol knowledge benchmark for AI models","tags":["ethiq","ai","ethereum","benchmark"],"image":"/img/blog/introducing-ethiq/performance-chart.png"},"unlisted":false,"prevItem":{"title":"Validator Report: Investigate Validator Performance","permalink":"/posts/validator-report"},"nextItem":{"title":"EVM Gas Profiling: New Execution Trace Data","permalink":"/posts/evm-gas-profiling"}},"content":"export const API = ({children}) => <span style={{color: \'#4a7c59\', fontWeight: \'bold\'}}>{children || <><span style={{display: \'inline-block\', width: \'0.55em\', height: \'0.55em\', borderRadius: \'50%\', backgroundColor: \'#4a7c59\', marginRight: \'0.25em\', verticalAlign: \'middle\'}} />API</>}</span>;\\nexport const Agentic = ({children}) => <span style={{color: \'#7c4a6e\', fontWeight: \'bold\'}}>{children || <><span style={{display: \'inline-block\', width: \'0.55em\', height: \'0.55em\', borderRadius: \'50%\', border: \'2px solid #7c4a6e\', marginRight: \'0.25em\', verticalAlign: \'middle\'}} />Agentic</>}</span>;\\n\\nIf you ask an AI model about consensus constants, EVM execution, or state transitions, does it actually know what it\'s talking about? We built [EthIQ](https://ethiq.ethpandaops.io), a benchmark that measures how well AI models understand Ethereum protocol internals.\\n\\nModels are tested in two modes: <API /> (direct API calls with a system prompt, no tools) and <Agentic /> (CLI tools like Claude Code and Codex running in sandboxed Docker containers with bash, file I/O, and Node.js).\\n\\n## Motivation\\nLLMs are now a daily tool at ethPandaOps, and we needed a way to evaluate them in our specific context. EthIQ gives us a quick signal when a new model releases, and a meaningful eval suite to drive improvements as we invest more in agentic tooling (prompt optimization, fine-tuning, workflows, etc.) \\n\\n## Questions\\n\\nThe question set is designed to stress two distinct model capabilities: world knowledge (tested by categories like Constants) and raw reasoning (tested by auto-generated categories like EVM Execution). World knowledge questions are intentional. We want to explicitly probe what Ethereum protocol knowledge made it into a model\'s training data, since recall of these values is genuinely useful.\\n\\nTo prevent memorization in the raw reasoning tasks, auto-generated questions start from official Ethereum spec test fixtures but mutate the inputs with a randomized seed (balances, storage values, calldata, deposit amounts, etc.) Ground-truth answers are then re-derived from the mutated inputs using the Python reference implementations. If a model saw the original fixtures during training, its answers won\'t match. New forks get a fresh seed, keeping the benchmark honest over time.\\n\\nQuestions are organized into datasets tied to Ethereum forks. The first dataset is **fusaka**, with 325 questions across these categories:\\n\\n| Category | What it asks | Example |\\n|:---|:---|:---|\\n| Constants | Exact protocol constant values | What is the value of `SLOTS_PER_EPOCH` on Mainnet? |\\n| EVM execution | Trace through bytecode, report the outcome | Given `0x...`, what\'s in storage slot 0x1? |\\n| Consensus state transitions | Apply slashings, deposits, etc. and compute resulting state | After this attester slashing, what is validator 6\'s balance? |\\n| Consensus epoch processing | Calculate rewards, penalties, and balance deltas | What are the reward/penalty deltas after processing this epoch? |\\n| Consensus fork choice | Replay block trees and determine the canonical head | After these attestations, which block
1is the head? |\\n| Consensus shuffling | Compute validator committee assignments | Which validators are in committee index 2 at slot 4? |\\n| Calculations | Multi-step arithmetic using protocol constants | How many seconds are in one Ethereum epoch? |\\n| Conceptual | Open-ended explanations graded by LLM rubric | Explain how RANDAO bias works in validator shuffling |\\n| Cross-fork | What changed between forks | What changed about max effective balance in Electra? |\\n| EIP interactions | How specific EIPs interact with each other | How do EIP-4844 blob commitments appear in beacon blocks? |\\n| Trick | Questions with wrong premises | \\"When did Verkle Trees ship on mainnet?\\" (They didn\'t.) |\\n\\n## The Results\\n\\n<ClickableImage\\n src=\\"/img/blog/introducing-ethiq/performance-chart.png\\"\\n alt=\\"EthIQ performance chart showing model pass rates over time\\"\\n caption=\\"Figure 1: Model performance by release date on the fusaka dataset (325 questions). Whiskers show 95% confidence intervals.\\"\\n/>\\n\\n### Frontier\\nIn <API /> mode, Anthropic\'s `claude-opus-4.6-high` leads at 77.5%, followed by OpenAI\'s `gpt-5.3-codex-xhigh` at 76.3% and Google\'s `gemini-3.1-pro-preview-high` at 72.3%. `claude-sonnet-4.6-high` is still in progress.\\n\\nWhen <Agentic /> runs are included, `gpt-5.3-codex-xhigh` takes the lead at 83.7% via the Codex CLI.\\n\\nWe observed some models ([`minimax-m2.5-high`](https://ethiq.ethpandaops.io/model/minimax-m2.5-high?dataset=fusaka)) suffering degraded performance as their `reasoning_effort` increased. After investigation, we found that these models were exhausting their maximum token output allocation, effectively thinking themselves to death.\\n\\n[EVM execution questions in <API /> mode](https://ethiq.ethpandaops.io/performance?dataset=fusaka&prompt=bare&categories=evm-execution) are a particularly good vibe check for model capability. Watching `Kimi K2.5` step through executing the EVM in it\'s thinking traces is quite an experience (read: concerning!) We felt bad asking `llama-3.2-1b` to do the same. In general, we weren\'t expecting models to be so capable at executing the EVM in-context. Fortunately we capped the difficulty of the EVM questions at generation time (based on a few heuristics), so once this dataset set begins to saturate we can raise these limits and unleash a new `very hard` class of questions.\\n\\n### Open Weights\\n<ClickableImage\\n src=\\"/img/blog/introducing-ethiq/openweights.png\\"\\n alt=\\"EthIQ performance chart showing model pass rates over time for only open weight models\\"\\n caption=\\"Figure 2: Open Weights model performance by release date on the fusaka dataset (325 questions). Whiskers show 95% confidence intervals.\\"\\n/>\\n\\nEthereum and open weights models go hand-in-hand, so we added the ability to [show just open weight models](https://ethiq.ethpandaops.io/performance?dataset=fusaka&prompt=bare&open_weights=true). Kimi\'s `k2.5-high` is the stand out amongst the open weight models, scoring 60.6%.\\n\\n`minimax-m2.5-high` was disappointing, landing at 37.4%. This bulk of this disparity is in `Consensus Constants`, with `k2.5-high` at 94.7% compared to `minimax-m2.5-high` at 58.9%. `k2.5` is a much larger model at 1 trillion parameters versus `minimax-m2.5` at 230 billion. World knowledge is an important factor when using an LLM for Ethereum!\\n\\n\\n### The Canary :bird:\\n[Thankfully all <API /> evaluations returned failures for `Consensus shuffling`](https://ethiq.ethpandaops.io/performance?dataset=fusaka&prompt=bare&categories=consensus-shuffling). This would require the models to compute SHA256 in-context. If this canary dies you\'ll find the ethPandaOps team in a remote location far away from any electricity.\\n\\n## Try it\\n\\nBrowse the full results at [ethiq.ethpandaops.io](https://ethiq.ethpandaops.io). You can filter by question category, difficulty, and evaluation mode. \\n\\nKeep an eye out as we\'ll be updating this as new forks ship and models are released!"},{"id":"evm-gas-profiling","metadata":{"permalink":"/posts/evm-gas-profiling","source":"@site/blog/2026-02-11-evm-gas-profiling/index.md","title":"EVM Gas Profiling: New Execution Trace Data","description":"New EVM execution trace data in the Xatu dataset, a Gas Profiler in The Lab, and a Gas Repricing Simulator for testing gas schedule changes against real blocks.","date":"2026-02-11T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"execution-layer","permalink":"/posts/tags/execution-layer"},{"inline":true,"label":"gas","permalink":"/posts/tags/gas"},{"inline":true,"label":"evm","permalink":"/posts/tags/evm"},{"inline":true,"label":"lab","permalink":"/posts/tags/lab"},{"inline":true,"label":"data","permalink":"/posts/tags/data"}],"readingTime":4.85,"hasTruncateMarker":false,"authors":[{"name":"matty","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/mattevans","github":"https://github.com/mattevans","twitter":"https://x.com/mattevansnz","imageURL":"/img/team/matty.jpeg","key":"matty","page":null},{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null}],"frontMatter":{"slug":"evm-gas-profiling","title":"EVM Gas Profiling: New Execution Trace Data","authors":["matty","savid"],"description":"New EVM execution trace data in the Xatu dataset, a Gas Profiler in The Lab, and a Gas Repricing Simulator for testing gas schedule changes against real blocks.","tags":["xatu","execution-layer","gas","evm","lab","data"],"image":"/img/blog/evm-gas-profiling/header.png","relatedLinks":[{"name":"Gas Profiler","url":"https://lab.ethpandaops.io/ethereum/execution/gas-profiler"},{"name":"Gas Repricing Simulator","url":"https://lab.ethpandaops.io/ethereum/execution/gas-profiler/simulate"},{"name":"Xatu Dataset","url":"/data/xatu"}],"githubRepos":[{"name":"xatu-cbt","url":"https://github.com/ethpandaops/xatu-cbt"},{"name":"lab","url":"https://github.com/ethpandaops/lab"}]},"unlisted":false,"prevItem":{"title":"Introducing EthIQ","permalink":"/posts/introducing-ethiq"},"nextItem":{"title":"Fusaka: More Blobs, Less Votes, Broken Hearts?","permalink":"/posts/fusaka-more-blobs-less-votes"}},"content":"import Alert from \'@site/src/components/Alert\';\\n\\nEthereum transactions trigger a sequence of EVM opcodes, each consuming gas according to the protocol\'s gas schedule. If you wanted to understand this execution at scale, across all transactions and blocks, you\'d need an archive node and a lot of patience.\\n\\nWe\'ve done that work for you. The [Xatu dataset](/data/xatu) now includes full EVM execution trace data: per-opcode gas consumption, call frame hierarchies, and pre-aggregated analytics across 9 new tables. The data starts at block `24,120,000` and updates continuously through `HEAD`.\\n\\nOn top of this data, we built two tools in [The Lab](https://lab.ethpandaops.io/ethereum/execution/gas-profiler):\\n\\n- A **Gas Profiler** for inspecting opcode-level gas consumption at the transaction, block, and network level\\n- A **Gas Repricing Simulator** that re-executes blocks with modified gas schedules, so you can test \\"what if\\" scenarios for gas repricing proposals\\n\\
1n<center>\\n\\n[View Gas Profiler](https://lab.ethpandaops.io/ethereum/execution/gas-profiler) | [View Gas Repricing Simulator](https://lab.ethpandaops.io/ethereum/execution/gas-profiler/simulate)\\n\\n</center>\\n\\n## The Data Pipeline\\n\\nThe raw data comes from `debug_traceTransaction` structlogs: step-by-step EVM execution traces recording each opcode, its gas cost, stack depth, and error status. We run this against every finalized transaction and aggregate the results into `canonical_execution_transaction_structlog_agg`, which stores per-frame, per-opcode summaries.\\n\\nFrom there, [CBT](https://github.com/ethpandaops/cbt) <InlineLogo src=\\"/img/cbt.png\\" alt=\\"CBT\\" /> transforms this raw structlog data through several layers:\\n\\n```mermaid\\ngraph TD\\n A[\\"canonical_execution_transaction_structlog_agg\\"] --\x3e B[\\"int_transaction_call_frame\\"]\\n A --\x3e C[\\"int_transaction_call_frame_opcode_gas\\"]\\n C --\x3e D[\\"int_transaction_opcode_gas\\"]\\n D --\x3e E[\\"int_block_opcode_gas\\"]\\n E --\x3e F[\\"fct_opcode_gas_by_opcode<br/>daily / hourly\\"]\\n E --\x3e G[\\"fct_opcode_ops<br/>daily / hourly\\"]\\n```\\n\\nEach layer builds on the previous one. The intermediate tables (`int_`) feed into the fact tables (`fct_`), which power the time-series views in The Lab. All of them are queryable directly in the [Xatu dataset](/data/xatu).\\n\\n## New Tables\\n\\n<Alert>\\nFull schema documentation: [Source table](/data/xatu/datasets/canonical_execution_/#canonical_execution_transaction_structlog_agg) | [CBT tables](/data/xatu/datasets/cbt/)\\n</Alert>\\n\\n| Table | Description |\\n|-------|-------------|\\n| [int_transaction_call_frame](/data/xatu/datasets/cbt/#int_transaction_call_frame) | Call tree structure for each transaction. Each row is a call frame with self gas, cumulative gas, intrinsic gas, refunds, call type, target address, and function selector. |\\n| [int_transaction_call_frame_opcode_gas](/data/xatu/datasets/cbt/#int_transaction_call_frame_opcode_gas) | Per-frame opcode breakdown. Which opcodes a specific call frame executed and how much gas each consumed. |\\n| [int_transaction_opcode_gas](/data/xatu/datasets/cbt/#int_transaction_opcode_gas) | Opcode aggregation at the transaction level. Total count and gas per opcode across all frames in a transaction. |\\n| [int_block_opcode_gas](/data/xatu/datasets/cbt/#int_block_opcode_gas) | Block-level opcode aggregation. Each row is one opcode\'s usage within a single block: count, gas, and error count. |\\n| [fct_opcode_gas_by_opcode_daily](/data/xatu/datasets/cbt/#fct_opcode_gas_by_opcode_daily) | Daily gas consumption ranked by opcode. Includes average gas per execution, block count, and error rates. |\\n| [fct_opcode_gas_by_opcode_hourly](/data/xatu/datasets/cbt/#fct_opcode_gas_by_opcode_hourly) | Hourly version of the above for finer-grained analysis. |\\n| [fct_opcode_ops_daily](/data/xatu/datasets/cbt/#fct_opcode_ops_daily) | Daily EVM throughput: ops/sec with p5/p50/p95 percentiles, Bollinger bands, and 7-day moving averages. |\\n| [fct_opcode_ops_hourly](/data/xatu/datasets/cbt/#fct_opcode_ops_hourly) | Hourly version with 6-hour moving averages. |\\n\\n### Data Availability\\n\\nEVM execution trace data is available from block `24,120,000` to `HEAD`. We\'re also backfilling earlier blocks. This will take a while given the volume of historical data, but we\'ll update coverage as it expands.\\n\\nThe intermediate and fact tables update incrementally, so new blocks show up shortly after finalization.\\n\\n## The Lab: Gas Profiler\\n\\nThe [Gas Profiler](https://lab.ethpandaops.io/ethereum/execution/gas-profiler) lets you explore gas data at three levels: **network**, **block**, and **transaction**.\\n\\nAt the network level, you can track opcode operations per second over time with statistical bands and see gas consumption broken down by opcode category.\\n\\nDrill into a block to see per-transaction gas breakdowns and opcode heatmaps. Go deeper into a transaction to inspect the full call tree and opcode-level gas distribution, all the way down to a specific `SSTORE` inside a nested `DELEGATECALL`.\\n\\n\\n\\n<div style={{textAlign: \'center\', margin: \'24px 0\'}}>
1\\n <a href=\\"https://lab.ethpandaops.io/ethereum/execution/gas-profiler\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'12px 24px\', backgroundColor: \'var(--ifm-color-primary)\', color: \'white\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'bold\'}}>View Gas Profiler \u2197</a>\\n</div>\\n\\n## The Lab: Gas Repricing Simulator\\n\\nThe [Gas Repricing Simulator](https://lab.ethpandaops.io/ethereum/execution/gas-profiler/simulate) re-executes blocks with a modified gas schedule, running the actual EVM with your custom costs. Pick a block (or a range of up to 50), override any combination of opcode costs and dynamic gas parameters (storage access, call costs, memory expansion, logging, hashing, etc.), and see what changes.\\n\\nThe simulator tells you which transactions would use more or less gas, which would exceed the block gas limit, and which would flip between success and failure. It runs real EVM execution, so contracts that check `gasleft()` may take different code paths under modified costs. The simulator flags these as diverged transactions.\\n\\nSay you want to know what happens if `SLOAD_COLD` goes from 2100 to 3000. Instead of reasoning about it abstractly, you can run it against recent blocks and see exactly which transactions break.\\n\\n\\n\\n<div style={{textAlign: \'center\', margin: \'24px 0\'}}>\\n <a href=\\"https://lab.ethpandaops.io/ethereum/execution/gas-profiler/simulate\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'12px 24px\', backgroundColor: \'var(--ifm-color-primary)\', color: \'white\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'bold\'}}>View Gas Repricing Simulator \u2197</a>\\n</div>\\n\\n## Wrapping up\\n\\nAll 9 tables are live in the [Xatu dataset](/data/xatu). The [Gas Profiler](https://lab.ethpandaops.io/ethereum/execution/gas-profiler) and [Gas Repricing Simulator](https://lab.ethpandaops.io/ethereum/execution/gas-profiler/simulate) are in The Lab.\\n\\nIf you have questions or ideas for what to build on this data, find us on [Twitter](https://x.com/ethpandaops) or in the [Xatu Telegram Group](https://t.me/+JanoQFu_nO8yNzQ1).\\n\\nLove,\\n\\nEthPandaOps Team"},{"id":"fusaka-more-blobs-less-votes","metadata":{"permalink":"/posts/fusaka-more-blobs-less-votes","source":"@site/blog/2026-01-15-more-blobs-less-votes/index.md","title":"Fusaka: More Blobs, Less Votes, Broken Hearts?","description":"Investigating how attestation performance changes as blobs change through Fusaka, BPO1, and BPO2.","date":"2026-01-15T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"data","permalink":"/posts/tags/data"},{"inline":true,"label":"timing-games","permalink":"/posts/tags/timing-games"},{"inline":true,"label":"blobs","permalink":"/posts/tags/blobs"}],"readingTime":4.92,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"fusaka-more-blobs-less-votes","title":"Fusaka: More Blobs, Less Votes, Broken Hearts?","authors":["samcm"],"description":"Investigating how attestation performance changes as blobs change through Fusaka, BPO1, and BPO2.","tags":["xatu","data","timing-games","blobs"],"image":"/img/blog/late-blocks-lost-votes/hero.jpg","relatedLinks":[{"name":"Miga Labs","url":"https://www.migalabs.io/"},{"name":"Xatu","url":"/data/xatu"},{"name":"Chainbound Analysis","url":"https://research.chainbound.io/peerdas-propagation#summary-of-results"},{"name":"Miga Labs Analysis","url":"https://ethresear.ch/t/blob-analysis-after-fusaka-and-bpo-updates/23853"}]},"unlisted":false,"prevItem":{"title":"EVM Gas Profiling: New Execution Trace Data","permalink":"/posts/evm-gas-profiling"},"nextItem":{"title":"Our Roadmap for 2026","permalink":"/posts/ethpandaops-in-2026"}},"content":"export const Conservative = ({children}) => <span style={{color: \'#2ecc71\', fontWeight: \'bold\'}}>{children || \'Conservative\'}</span>;\\nexport const Neutral = ({children}) => <span style={{color: \'#f39c12\', fontWeight: \'bold\'}}>{children || \'Neutral\'}</span>;\\nexport const Suspect = ({children}) => <span style={{color: \'#e74c3c\', fontWeight: \'bold\'}}>{children || \'Suspect\'}</span>;\\n\\nWith [BPO2](https://lab.ethpandaops.io/ethereum/forks/bpo2) out the door, it\'s time to see how the network is handling the increased blob count. This analysis uses 1 month of block arrival data from [Xatu](/data/xatu). 147 nodes across 29 countries and 73 cities contributed timing data, many of them run by community members through our [public contributor program](/contribute-data/). This data is combined with the excellent validator data from [Miga Labs](https://www.migalabs.io/) to identify validator entities.\\n\\nThe first chart of this analysis expedition was very concerning:\\n\\n<ClickableImage\\n src=\\"/img/blog/late-blocks-lost-votes/hook-chart.png\\"\\n alt=\\"Attestation Head Votes vs Blob Count\\"\\n caption=\\"Figure 1: Attestation head votes (%) against blob count. 1 month of Mainnet data.\\"\\n/>\\n<center><h2 style={{marginTop: \'2rem\', marginBottom: \'2rem\'}}>wtf?</h2></center>\\n\\nWe define head votes here as \\"attestations that voted for the block that was proposed in the same slot\\". The initial results show a clear drop in head votes as blob count rises.\\n\\nObviously this isn\'t good for our future scaling plans! Could this be the limit of the network? Was this the last ever BPO fork?! :scream:\\n\\nAdding more blobs means more data to publish within the same time window (4s). Combining this with the additional data for attesters to download and verify. Some degradation was expected.. but definitely not this much. Excellent analysis from [Chainbound](https://research.chainbound.io/peerdas-propagation#summary-of-results) and [Miga Labs](https://ethresear.ch/t/blob-analysis-after-fusaka-and-bpo-updates/23853) also found similar results. \\n\\nWe need to investigate further.\\n\\nCould timing games be the cause? Timing games are when proposers intentionally delay the publication of their blocks to extract as much MEV as possible. For more information, check out Toni\'s [ethresear.ch post](https://ethresear.ch/t/on-attestations-block-propagation-and-timing-games/20272).\\n\\n\\
1n## Methodology\\n\\nTo investigate, proposer entities were categorized by their timing behavior:\\n\\n1. **Calculate a Timing Game Score** for each entity by looking at the median first-seen block time across their proposed slots\\n2. **Bucket entities** into three categories:\\n - <Conservative />: Score below 0 - blocks arrive early\\n - <Neutral />: Score between 0 and 2\\n - <Suspect />: Score above 2 - blocks consistently arrive late\\n3. **Calculate attestation head vote %** for slots proposed by each category\\n\\n### Score Distribution\\n\\n<ClickableImage\\n src=\\"/img/blog/late-blocks-lost-votes/score-distribution.png\\"\\n alt=\\"Timing Game Score Distribution\\"\\n caption=\\"Figure 2: Distribution of timing game scores across 564 entities. Most entities are conservative (346), with a small tail of suspects (36).\\"\\n/>\\n\\nMost entities fall into the <Conservative /> bucket (346 entities). 182 are <Neutral />, and only 36 are flagged as <Suspect />. The red dashed line marks the suspect threshold at score > 2.\\n\\n### Timing Scores by Entity\\n\\n<ClickableImage\\n src=\\"/img/blog/late-blocks-lost-votes/timing-scores.png\\"\\n alt=\\"Timing Game Score by Entity\\"\\n caption=\\"Figure 3: Top 60 entities by validator count, sorted by timing game score.\\"\\n/>\\n\\nThis shows the top 60 entities by validator count, sorted by their timing game score. A clear gradient runs from <Conservative>conservative</Conservative> entities at the top to <Suspect>suspects</Suspect> at the bottom. A <Suspect /> entity here doesn\'t necessarily mean they\'re a bad actor or intentionally playing timing games, it just means their blocks are consistently arriving late to the Xatu nodes.\\n\\n## Entity Performance Overview\\n\\n<ClickableImage\\n src=\\"/img/blog/late-blocks-lost-votes/entity-performance.png\\"\\n alt=\\"Entity Performance Scatter Plot\\"\\n caption=\\"Figure 4: Entity performance scatter plot. X-axis: median block arrival time (ms). Y-axis: head vote percentage (%). Bubble size: validator count.\\"\\n/>\\n\\nEach dot is an entity. The cluster of <Conservative>green dots</Conservative> on the left side (early block arrivals) all sit at ~99-100% votes. As you move right into late-arriving blocks, votes for the proposer\'s block drop.\\n\\nWith this relationship established, we can now look at how blob count affects each timing category.\\n\\n## Results\\n\\n<ClickableImage\\n src=\\"/img/blog/late-blocks-lost-votes/blob-impact.png\\"\\n alt=\\"Blob Count Impact by Timing Category\\"\\n caption=\\"Figure 5: Head vote percentage (%) vs blob count, grouped by timing game category.\\"\\n/>\\n\\n**The <Conservative>green line</Conservative> stays flat at ~99% regardless of blob count**. I breathed a sigh of relief here. This is a good sign!\\n\\n<Conservative /> entities that publish blocks promptly maintain near-perfect attestation head votes even at 21 blobs. The decay observed in the aggregate chart is almost entirely driven by entities playing timing games.\\n\\nThe <Suspect /> group (<Suspect>red line</Suspect>) starts at 98% with 1 blob and drops to 92% at 21 blobs. <Neutral /> falls somewhere in between.\\n\\nThis makes sense: if blocks are already being published late, adding more blobs makes an already tight timing window even harder to hit.\\n\\n## Data Notes\\n\\n- **Time period**: 1 month\\n- **Exclusions**:\\n - Slots with less than 5% attestation rate (likely infrastructure issues rather than timing problems)\\n - Time periods where there were known issues with clients that were unrelated to the analysis.\\n\\n\\n## Takeaways\\n\\n1. **Timing games, not blob count, are the primary driver of attestation degradation.**\\n\\n<Conservative /> entities maintain ~99% head votes even at max blob counts. \\n\\n2. **Timing games proposers aren\'t going anywhere.**.\\n\\nEven though we\'re seeing a clear degradation in head votes, the <Suspect /> entities are still able to maintain a high enough head vote percentage to keep their blocks in the chain. *Core devs and the community need to account for this in future planning.* We can\'t ignore these entities.\\n\\n3. **Pro
1posing a block is getting harder**.\\n\\nAll large operators should be monitoring their infrastructure closely.\\n\\nWhile this analysis is a good starting point, it doesn\'t cover the full picture for justifying the next BPO values. More data and analysis is needed. We\'ll continue to monitor the network and publish more analysis as we go."},{"id":"ethpandaops-in-2026","metadata":{"permalink":"/posts/ethpandaops-in-2026","source":"@site/blog/2026-1-09-ethpandaops-in-2026/index.md","title":"Our Roadmap for 2026","description":"A look at what\'s coming up for ethPandaOps in 2026!","date":"2026-01-09T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"data","permalink":"/posts/tags/data"},{"inline":true,"label":"execution","permalink":"/posts/tags/execution"},{"inline":true,"label":"l2","permalink":"/posts/tags/l-2"},{"inline":true,"label":"roadmap","permalink":"/posts/tags/roadmap"},{"inline":true,"label":"l1","permalink":"/posts/tags/l-1"}],"readingTime":10.89,"hasTruncateMarker":false,"authors":[{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"matty","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/mattevans","github":"https://github.com/mattevans","twitter":"https://x.com/mattevansnz","imageURL":"/img/team/matty.jpeg","key":"matty","page":null},{"name":"pk910","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/pk910","github":"https://github.com/pk910","twitter":"https://x.com/_pk910_","imageURL":"/img/team/pk910.jpeg","key":"pk910","page":null},{"name":"skylenet","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/skylenet","github":"https://github.com/skylenet","twitter":"https://x.com/skylenet","imageURL":"/img/team/skylenet.jpeg","key":"skylenet","page":null},{"name":"qu0b","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/qu0b","github":"https://github.com/qu0b","twitter":"https://x.com/stefan_star","imageURL":"/img/team/qu0b.jpeg","key":"qu0b","page":null},{"name":"barnabasbusa","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/barnabasbusa","github":"https://github.com/barnabasbusa","twitter":"https://x.com/BarnabasBusa","imageURL":"/img/team/barnabasbusa.jpeg","key":"barnabasbusa","page":null},{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null}],"frontMatter":{"slug":"ethpandaops-in-2026","title":"Our Roadmap for 2026","authors":["savid","samcm","matty","pk910","skylenet","qu0b","barnabasbusa","parithosh"],"description":"A look at what\'s coming up for ethPandaOps in 2026!","tags":["xatu","data","execution","l2","roadmap","l1"],"image":"/img/blog/roadmap-2026/factorio-panda.png","githubRepos":[{"name":"ethPandaOps","url":"https://github.com/ethpandaops"}],"relatedLinks":[{"name":"The Lab","url":"https://lab.ethpandaops.io/"},{"name":"Xatu Dataset","url":"/data/xatu"}]},"unlisted":false,"prevItem":{"title":"Fusaka: More Blobs, Less Votes, Broken Hearts?","permalink":"/posts/fusaka-more-blobs-less-votes"},"nextItem":{"title":"Announcing Live Custody Monitoring for PeerDAS","permalink":"/posts/live-custody-monitoring-peerdas"}},"content":"We had someone come up to us at [Devconnect 2025](https://devconnect.org/) and ask us what exactly the team does, especially since we tend to be everywhere. That was a surprising moment, but it showed that we need to do a better job of communicating why the team works on what it does! The easiest explanation is: \\"Whatever Ethereum needs to ship forks securely and quickly\\". However, this is a vague statement. What does this actually entail? We\'ll try our best to explain ourselves while outlining what we aim to tackle in 2026!\\n\\nIt\'s been a huge year for the [ethPandaOps](https://ethpandaops.io/) team, We expanded our ability to ship forks, help host [ACDT](https://ethereum-magicians.org/t/all-core-devs-acd/24198). revamped tooling, grew our data pipeline and set the stage for a productive year! The most obvious part of shipping a fork is co-ordination. A portion of the team is actively involved in herding EIPs through the process of acceptance on [ACD](https://ethereum-magicians.org/t/all-core-devs-acd/24198), to prototypes, to interoperable client implementations and finally hardening the implementation to go live on Mainnet via testing. This pipeline of shipping EIPs requires extensive tooling to test and understand the feature before it goes live, hence a portion of the team works on tooling. The final portion of the team works on data, this is data that we feed into the first two portions - to understand EIPs, the effects of tests and to understand their implications for Mainnet. By tightly integrating these three portions, we\'re always able to have quick feedback loops for whatever changes core devs decide to throw our way.\\n\\nNow that we\'ve defined the three portions of the team, lets look a bit deeper into what\'s on our plate for the year:\\n## Maintaining all of our tooling!\\nAcross the fields of testing, network observability, data analytics, deployments and protocol development - we have a total of 30 tools. Not only do we maintain these tools, we also often contribute to their upstream dependencies to keep up with forks. These tools are critical for us to continue supporting Ethereum, they will always remain first priority for us. You can find an exhaustive list of tooling [here](https://ethpandaops.io/docs/tooling/overview/).\\n\\n## Glamsterdam and Hegota\\nGlamsterdam is the next fork on Ethereum and is almost done finalizing scope. We are actively working on [Block Level Access lists](https://ethereum-magicians.org/t/eip-7928-block-level-access-lists/23337) and ramping up our understanding of [ePBS](https://ethereum-magicians.org/t/eip-7732-enshrined-proposer-builder
1-separation-epbs/19634). We are currently updating our tooling to best visualize BALs and in the future will focus on tooling for test cases in ePBS. In order to do 2 forks a year, we need to start pipelining our efforts. We intend to start pushing for prototypes and figuring out testing as soon as possible to ensure we have a smooth pipeline with reasonable timelines. You can find some experiments using AI to speed up our debugging process in our [ai-cookbook](https://github.com/ethpandaops/ai-cookbook).\\n\\n## Tooling for Scaling\\nOver the last year, Ethereum has managed to scale the [gas throughput by 2x](https://etherscan.io/chart/gaslimit) as well as the [DA layer by 3.5x](https://dune.com/queries/3757544/6319515). There are already plans for rapid scaling throughout the course of the year - both on the Gas and DA scaling fronts. \\n\\nIn order to help increase the gas limit, We [collaborated with the Nethermind team](https://www.nethermind.io/blog/measuring-ethereums-execution-limits-the-gas-benchmarking-framework), organized the [scaling efforts](https://ethpandaops.io/posts/gaslimit-scaling/), [planned the rollout](https://ethpandaops.io/posts/60m-gas-sepolia-hoodi/) and [analyzed the change on Mainnet](https://lab.ethpandaops.io/ethereum/execution/timings). While the [Nethermind gas benchmark](https://github.com/NethermindEth/gas-benchmarks) tool has been invaluable this year, we notice some room for improvement as the scope for the tool grew. We plan on addressing this by improving the benchmark framework itself. The idea is to generalize the benchmark framework such that it integrates neatly with [EEST](https://eest.ethereum.org/main/) while being used by the [State research](https://cperezz.github.io/bloatnet-website/index.html), [Repricing initiative](https://blog.base.dev/glamsterdam-proposals#h-gas-repricing), [Scale L1](https://blog.ethereum.org/2025/08/05/protocol-update-001) and [zkVM](https://zkevm.ethereum.foundation/blog/benchmarking-zkvms) teams. This mostly involves making the tool easier to run locally, in a CI, on demand and with forks - all with minimal configuration. We additionally want to make this tool easier to contribute to, hopefully enabling [L2 collaboration](https://blog.base.dev/scaling-base-in-2025) in scaling the EVM!\\n\\nWith the unbundling of DA increases from forks via [BPOs](https://etherworld.co/bpo-forks-explained-how-fusaka-gradually-scales-blob-capacity/), we intend to work with the [p2p team](https://github.com/ethp2p/) to ship Partial cell proofs. This feature would allow for more scaling while allowing for a higher degree of reuse of blobs already in the mempool. This effort will be tied tightly to the data collection front, signaling when we are ready to scale. \\n\\n<ClickableImage\\nsrc=\\"/img/blog/bpo-feedback-loop.jpeg\\"\\nalt=\\"BPO feedback loop\\"\\ncaption=\\"The BPO feedback loop: from Contributoor data to network analysis\\"\\nstyle={{maxWidth: \'600px\'}}\\n/>\\n\\n## Doubling down on Xatu\\n[Xatu](https://github.com/ethpandaops/xatu). What started as an internal tool in December 2022 has evolved into a full-fledged data pipeline that is now open source and available to the community. We [open-sourced the dataset](/posts/open-source-xatu-data/) in March 2024 and since then we\'ve been collecting and transforming data from multiple Ethereum networks. Xatu is going to continue to expand its scope over the year, acting as a central data lake for all of our endeavours.\\n\\nOur data stack has historically been focused on the consensus layer. We intend to change this! We intend to look into the following topics this year, while also integrating them into the tracing and scaling efforts:\\n**State Analysis**: Understanding state growth patterns and access frequency, Informing state expiry discussions with real data \\n**Engine API Timings**: Measuring how long execution clients take to process blocks, Identifying bottlenecks and performance regressions \\n**Transaction Analysis**: Deeper mempool flow analysis, Transaction propagation timing across the network \\n**Contract Analysis**: OPCODE and precompile usage patterns,Gas consumption profiles across different c
1ontract types \\n\\nAdditionally we wish to start collecting data from L2s to help educate decisions on L1 better:\\n- Understanding how Layer 2s consume blobspace\\n- Tracking blob markets and pricing dynamics\\n- Monitoring the health of Ethereum\'s role as the data layer\\n\\nThe long-term vision includes extending Xatu\'s reach to capture Layer 2 network data directly, providing the same level of visibility we have for Layer 1 across the broader Ethereum ecosystem.\\n\\n## Expanding The Lab\\n\\nWe [launched the lab](/posts/the-lab/) in November to be the first stop for understanding Ethereum\'s protocol. This relaunch converted a side project into a full fledged member of our toolchain. This year, you can expect a lot more from the Lab: Execution layer views, State analysis, Data availability monitoring, and Live fork visualization. Additionally, If you can see it in [the lab](https://lab.ethpandaops.io/), you\'ll be able to query it in the [Xatu dataset](/data/xatu) - enabling any reearcher to dig into theories. The data that comes out of the lab also helps us understand the effects of scaling Ethereum, allowing a strict feedback loop in our process.\\n\\nThe [CBT pipeline](/posts/the-lab/#the-power-of-cbt-) (explained below) makes this possible. New models, new tables, new visualizations - the full stack from data transformation to API is automated. We\'ll be shipping curated views of the data, not just raw dumps.\\n\\n## Data Pipeline Evolution\\n\\n[CBT (ClickHouse Build Tool)](https://github.com/ethpandaops/cbt) lets us compose interconnected tables that build on each other. The [xatu-cbt](https://github.com/ethpandaops/xatu-cbt) project currently contains 115 transformation models that power [the lab](https://lab.ethpandaops.io/), and we\'re working towards making them all available to the community in 2026.\\n\\nThese tables are faster to query than our raw event tables and more directly useful for common analysis patterns. Whether you\'re looking at attestation correctness, block propagation, or custody compliance - there\'s likely a pre-computed table that gets you there faster.\\n\\nThe pipeline is fully automated: code generation from ClickHouse schemas to Protobuf definitions to OpenAPI specs. When we add a new model, the entire stack updates.\\n\\n## Open Data Infrastructure\\n\\nWe believe in [open data](https://en.wikipedia.org/wiki/Open_data) - all Xatu data is released under [CC BY 4.0](https://github.com/ethpandaops/xatu-data) - and we want to make it as accessible as possible. In 2026, we\'re expanding how you can access and work with Xatu data.\\n\\nWe\'re making more data available as Parquet files for local querying and research. This includes the CBT tables that power [the lab](https://lab.ethpandaops.io/) - the derived views, not just raw events. We\'re also exploring [Apache Iceberg](https://iceberg.apache.org/) data catalogs to make discovery easier - exposing a standard REST catalog interface so you can connect engines you already use, like Spark, Snowflake, and PyIceberg.\\n\\nFor those who want full data sovereignty we\'re working to make the entire pipeline - Xatu collection, CBT transformations, and API layer - easily deployable. You can run your own infrastructure, collect your own data, and maintain complete control.\\n\\n## ZKVMs\\nThe world of ZKVM and Proving is approaching us quickly. We\'re working with the [zkvm](https://zkevm.ethereum.foundation/) team to self host infrastructure to allow us to understand bottlenecks holistically. The testing stack for zkvms have been separate till now, we\'ve started integrated the stack with our existing [Kurtosis ethereum-package](https://github.com/ethpandaops/ethereum-package). This will allow us to test out optional proofs locally and enables rapid iteration, while fitting into the existing testing stack.\\n\\n## Hardening\\nEthereum forks continue to increase the complexity of the Protocol. While we have some items in the roadmap intending to reduce the complexity, we need to still ensure that Ethereum today is hardened against attacks. We are working with the Lighthouse team at [sigmaPrime](https://sigmaprime.io/) to codify some attacks as well as fixes that don\'t require a fork to ensure a more resilient network. These are largely outcomes of the [Holesky incident](https://blog.sigmaprime.io/pectra-holesky-incident.html) as well as [Non-finality devnets](https://github.com/ethpandaops/nft-devnets) over the last year. We will additionally plan more attack networks throughout the year, allowing us to help surface more areas to focus on. While devnets help find one layer of issues, we also aim to supp
1ort the [TxRx](https://github.com/txrx-research) team in expanding the scope for [Compliance testing](https://github.com/ethereum/consensus-compliance-tests) to catch issues at the specification level.\\n\\n## Collaborations\\n\\nWe\'re building with teams across the EF and community:\\n\\n**[EF Stateless Consensus Team](https://x.com/StatelessEth)**\\nTogether we\'re exploring state access patterns and what they mean for Ethereum\'s stateless future. Some of the upcoming execution layer sections of [the lab](https://lab.ethpandaops.io/) are built in collaboration with this team.\\n\\n**[EF P2P Networking Team](https://x.com/raulvk)**\\nOur recent [custody monitoring launch](/posts/live-custody-monitoring-peerdas/) with [dasmon](https://github.com/ethp2p/dasmon) is just the beginning. As PeerDAS goes live, this collaboration will be critical for ensuring the network\'s data availability guarantees are being met.\\n\\n**[Scale L1](https://blog.ethereum.org/2025/08/05/protocol-update-001)**\\nWorking with the Scale L1 initiative on gas limit scaling, tracing, and benchmarking efforts to push network throughput.\\n\\n**[STEEL](https://steel.ethereum.foundation/)**\\nWorking with the STEEL team on all things testing related to the fork\\n\\n**[zkEVM](https://zkevm.ethereum.foundation/)**\\nWorking with the zkEVM team to ensure they get delivered safely\\n\\n**Client Teams**\\nExecution timing analysis, client diversity monitoring, and performance insights that help client teams optimize their implementations.\\n\\n**You?**\\nIf you\'re working on research, testing, tooling, or analysis that could benefit Ethereum - we\'d love to hear from you. Reach out via [Twitter](https://twitter.com/ethpandaops) or the [Xatu Telegram Group](https://t.me/+JanoQFu_nO8yNzQ1).\\n\\n## Thanks\\n\\nWe\'re extremely grateful for the continued support from the community that we\'ve received via engagement and the Contributoor program. This team does what it does only because of you.\\n\\n<div style={{textAlign: \'center\', margin: \'24px 0\'}}>\\n <div style={{display: \'inline-flex\', gap: \'12px\', flexWrap: \'wrap\', justifyContent: \'center\'}}>\\n <a href=\\"https://lab.ethpandaops.io/\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'10px 20px\', backgroundColor: \'var(--ifm-color-primary)\', color: \'white\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'500\', fontSize: \'0.9rem\'}}>Explore The Lab \u2197</a>\\n <a href=\\"/data/xatu/\\" style={{display: \'inline-block\', padding: \'10px 20px\', backgroundColor: \'transparent\', color: \'var(--ifm-color-primary)\', border: \'1px solid var(--ifm-color-primary)\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'500\', fontSize: \'0.9rem\'}}>Query Xatu Data \u2197</a>\\n <a href=\\"https://t.me/+JanoQFu_nO8yNzQ1\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'10px 20px\', backgroundColor: \'transparent\', color: \'var(--ifm-color-primary)\', border: \'1px solid var(--ifm-color-primary)\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'500\', fontSize: \'0.9rem\'}}>Join Xatu Telegram \u2197</a>\\n </div>\\n</div>\\n\\nWe\'re locking in for 2026. The Factory Must Grow. \\nYour ethPandaOps Team \u2764\ufe0f \\n\\n<ClickableImage\\nsrc=\\"/img/blog/roadmap-2026/the-factory-must-grow.png\\"\\nalt=\\"The Factory Must Grow\\"\\ncaption=\\"The Factory Must Grow\\"\\n/>"},{"id":"live-custody-monitoring-peerdas","metadata":{"permalink":"/posts/live-custody-monitoring-peerdas","source":"@site/blog/2025-12-03-live-custody-monitoring-peerdas/index.md","title":"Announcing Live Custody Monitoring for PeerDAS","description":"","date":"2025-12-03T00:00:00.000Z","tags":[{"inline":true,"label":"blob","permalink":"/posts/tags/blob"},{"inline":true,"label":"peerdas","permalink":"/posts/tags/peerdas"},{"inline":true,"label":"monitoring","permalink":"/posts/tags/monitoring"},{"inline":true,"label":"scaling","permalink":"/posts/tags/scaling"},{"inline":true,"label":"data-columns","permalink":"/posts/tags/data-columns"}],"readingTime":4.75,"hasTruncateMarker":false,"authors":[{"name":"raul","title":"P2P Networking","bio":"P2P Networking","url":"https://github.com/raulk","github":"https://github.com/raulk","imageURL":"/img/team/raul.jpeg","key":"raul","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","t
1witter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null},{"name":"matty","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/mattevans","github":"https://github.com/mattevans","twitter":"https://x.com/mattevansnz","imageURL":"/img/team/matty.jpeg","key":"matty","page":null}],"frontMatter":{"slug":"live-custody-monitoring-peerdas","title":"Announcing Live Custody Monitoring for PeerDAS","authors":["raul","samcm","savid","matty"],"description":"","tags":["blob","peerdas","monitoring","scaling","data-columns"],"image":"/img/blog/announcing-dasmon.png","relatedLinks":[{"name":"Live Custody Monitoring","url":"https://lab.ethpandaops.io/ethereum/data-availability/custody"},{"name":"dasmon","url":"https://github.com/ethp2p/dasmon"}]},"unlisted":false,"prevItem":{"title":"Our Roadmap for 2026","permalink":"/posts/ethpandaops-in-2026"},"nextItem":{"title":"Introducing The Lab","permalink":"/posts/the-lab"}},"content":"## Introduction\\n\\n<div style={{clear: \'right\', float: \'right\', width: \'40%\', textAlign: \'center\', marginTop: \'8px\', marginBottom: \'16px\', marginLeft: \'16px\'}}>\\n <div style={{display: \'flex\', flexDirection: \'column\', gap: \'8px\', alignItems: \'center\'}}>\\n <a href=\\"https://lab.ethpandaops.io/ethereum/data-availability/custody\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'10px 20px\', backgroundColor: \'var(--ifm-color-primary)\', color: \'white\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'500\', fontSize: \'0.9rem\', width: \'100%\', boxSizing: \'border-box\'}}>View Custody Compliance \u2197</a>\\n <a href=\\"https://lab.ethpandaops.io/ethereum/data-availability/probes?isLive=true&page=1\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'10px 20px\', backgroundColor: \'transparent\', color: \'var(--ifm-color-primary)\', border: \'1px solid var(--ifm-color-primary)\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'500\', fontSize: \'0.9rem\', width: \'100%\', boxSizing: \'border-box\'}}>View Live Custody Probes \u2197</a>\\n </div>\\n</div>\\n\\nFusaka entirely transforms Ethereum\'s data availability layer. For the first time, Ethereum moves from full replication, where every node stores every blob, to a sharded architecture where nodes store a variable subset of data columns depending on custody requirements. This is what makes the dramatic blob scalability improvements possible, gradually increasing max blob capacity from [9 to 72 by mid 2026](https://stokes.io/blob-scaling-fyi/), possibly even beyond.\\n\\nBut with this architectural shift comes a new challenge:\\n\\n> \\"how can users verify that the network is actually upholding its collective data availability guarantees?\\"\\n\\nToday, together with the P2P Networking team at the EF, we\'re launching a live custody monitoring tool hosted inside [The Lab](https://lab.ethpandaops.io/ethereum/data-availability/custody) to answer exactly that question, and several others, on a real-time basis.\\n\\n## The Sampling Problem\\n\\nIn the pre-Fusaka world, data availability was simple to reason about: if a node has a blob, its data is available. PeerDAS changes this logic fundamentally.\\n\\nNow, each beacon node commits to storing a partial view of the data, which is erasure coded to add 2x redundancy, and then split into 128 columns, sized 2KiB each. Each beacon node calculates its column custody set from its identity and effective validator balance. Sufficient column diversity is required to reconstruct the blob matrix attached to a block, within its custody window of 4096 epochs (~18 days). This threshold is 64 distinct columns.\\n\\nThese new dynamics trigger the need to continuously measure network-wide reliability to answer crucial questions in real time:\\n\\n- Are peers that claim to store certain columns indices actually storing them?\\n- Are peers successfully honouring their custody window?\\n- Are peers correctly serving data when requested?\\n- Are there any widespread compliance failures that could threaten data availability of certain blobs?\\n\\nWe can then close the loop by taking proactive actions when degradation is detected, such as alerting when custody for a particular blob matrix begins to approach a critical threshold, and initiating preemptive reconstruction and reseeding to improve availability ratios where needed.\\n\\n## Introducing Live Custody Monitoring\\n\\nTo provide visibility into these critical questions, we\'re launching a real-time monitoring dashboard that tracks custody compliance across the network. The dashboard is powered by [dasmon](https://github.com/ethp2p/dasmon), a custody monitoring engine developed by the P2P networking team at the Ethereum Foundation.\\n\\n\\n\\n**dasmon** continuously samples peers across the network, testing their claimed custody ranges by requesting data columns and validating the responses against known KZG commitments. Every test result is logged, analyzed, streamed to Xatu, and surfaced in real-time on the dashboard. The evolution of custody windows is also tracked on a peer-by-peer basis.\\n\\n## What dasmon does\\n\\
1nAt its core, dasmon answers a simple question: are peers storing the data they say they\'re storing?\\n\\nIt does this through continuous verification:\\n\\n1. **Peer tracking:** dasmon automatically discovers and tracks peers joining the network, monitoring their custody metadata, which columns they claim to store, and for which slot ranges.\\n2. **Intelligent sampling:** Rather than testing randomly, dasmon uses coverage-aware sampling to systematically verify untested (slot, column) combinations. This maximizes coverage while minimizing redundant testing.\\n3. **Response validation:** Every response is validated against known KZG commitments. dasmon verifies not just that data was returned, but that the data matches what should be stored.\\n4. **Real-time event streaming:** All test results, custody updates, and peer status changes are streamed in real-time to Xatu, enabling live monitoring dashboards and alerting like the one we\u2019re sharing today.\\n5. **Historical analytics:** the cumulative Xatu dataset enables long-term compliance analysis and trend detection.\\n\\n## Try it live\\n\\nThe monitoring dashboard is now live in [The Lab](https://lab.ethpandaops.io/ethereum/data-availability/custody). You can watch real-time custody compliance data flowing in as dasmon continuously verifies peers across the network.\\n\\nThis tool represents our commitment to making Ethereum\'s data availability layer not just scalable, but observable and verifiable. As the network transitions to PeerDAS, this visibility becomes essential for operators, researchers, and users who need confidence that the network\'s guarantees are being upheld.\\n\\n<div style={{textAlign: \'center\'}}>\\n <div style={{display: \'inline-flex\', flexDirection: \'column\', gap: \'8px\'}}>\\n <a href=\\"https://lab.ethpandaops.io/ethereum/data-availability/custody\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'12px 24px\', backgroundColor: \'var(--ifm-color-primary)\', color: \'white\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'bold\'}}>View Custody Compliance \u2197</a>\\n <a href=\\"https://lab.ethpandaops.io/ethereum/data-availability/probes?isLive=true&page=1\\" target=\\"_blank\\" style={{display: \'inline-block\', padding: \'12px 24px\', backgroundColor: \'transparent\', color: \'var(--ifm-color-primary)\', border: \'1px solid var(--ifm-color-primary)\', borderRadius: \'6px\', textDecoration: \'none\', fontWeight: \'bold\'}}>View Live Custody Probes \u2197</a>\\n </div>\\n</div>\\n\\n## Want to run a self-hosted dasmon?\\n\\ndasmon is open source and can be integrated into any Ethereum consensus client. For operators interested in running their own monitoring infrastructure, reference implementations and documentation are available on [Github](https://github.com/ethp2p/dasmon?tab=readme-ov-file#getting-started)."},{"id":"the-lab","metadata":{"permalink":"/posts/the-lab","source":"@site/blog/2025-11-14-the-new-lab/index.md","title":"Introducing The Lab","description":"Introducing the new The Lab platform","date":"2025-11-14T00:00:00.000Z","tags":[{"inline":true,"label":"data","permalink":"/posts/tags/data"},{"inline":true,"label":"lab","permalink":"/posts/tags/lab"}],"readingTime":6.69,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null},{"name":"matty","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/mattevans","github":"https://github.com/mattevans","twitter":"https://x.com/mattevansnz","imageURL":"/img/team/matty.jpeg","key":"matty","page":null}],"frontMatter":{"slug":"the-lab","title":"Introducing The Lab","authors":["samcm","savid","matty"],"description":"Introducing the new The Lab platform","tags":["data","lab"],"image":"/img/blog/the-new-lab/header.png","relatedLinks":[{"name":"The Lab","url":"https://lab.ethpandaops.io/"}]},"unlisted":false,"prevItem":{"title":"Announcing Live Custody Monitoring for PeerDAS","permalink":"/posts/live-custody-monitoring-peerdas"},"nextItem":{"title":"Fusaka bandwidth estimation","permalink":"/posts/fusaka-bandwidth-estimation"}},"content":"## Introduction\\n\\nAlmost a year ago, we quietly released an experimental version of The Lab with the goal of providing a public facing way to visualize the data we collect from the [Xatu](/data/xatu) <InlineLogo src=\\"/img/xatu.png\\" alt=\\"Xatu\\" /> project. We didn\'t really announce it anywhere, and were using it as a playground to experiment with new ideas. While it was a hit, it was limited in functionality and labor intensive to add new features to. As a result it has been sitting dormant for the last few months.\\n\\n<center>\\n**Today we\'re excited to officially launch a brand new version of The Lab.**\\n\\nCheck it out here: [lab.ethpandaops.io](https://lab.ethpandaops.io/)\\n</center>\\n\\n## The Lab\\n\\nWe set out to build a new version of The Lab with a single north star: **Make The Lab the community\'s first stop for verifying network performance.** While we aren\'t quite there yet, we are aiming to rapidly iterate and improve the platform to get it to that point as soon as possible. Today\'s release aims to be at-parity with the previous experimental version of The Lab.\\n\\nSince our team formed during The Merge, we\'ve built numerous dashboards and tools to help us understand Ethereum network internals. This data requires deep context to interpret that our team already possesses. The Lab bridges this gap, allowing us to provide curated views of the data that are easy to understand.\\n\\nAdditionally, **all the raw data that you see in The Lab will be directly available to the community.** Ethereum is built on the principles of openness and transparency, and we strongly believe in the power of open data. We are committed to continuing to build on these principles in the form of the [Xatu](/data/xatu) <InlineLogo src=\\"/img/xatu.png\\" alt=\\"Xatu\\" /> data project. For more info check out the [New Clickhouse Tables](#new-clickhouse-tables) section below.\\n\\nNote: The Lab aims to provide a different perspective on the network that is complementary to existing explorers or monitoring tools that the community already uses, like Dora, Beaconcha.in, Etherscan, etc.\\n\\n### A fresh coat of paint\\n\\nWe\'ve completely redesigned The Lab with the polish and performance of a modern web application. This includes a new design system, 3 new themes (Light, Dark, and Star) and a refreshed approach to the user experience.\\n\\n<ClickableImage\\n src=\\"/img/blog/the-new-lab/live.png\\"\\n alt=\\"The new Lab\\"\\n caption=\\"The Ethereum Live section within The Lab\\"\\n/>\\n\\n### Upcoming features\\nWe\'ll be adding a handful of new features to The Lab in the coming weeks. Some of these features are already in development but just need a little more time in the wok. Here\'s a sneak peek of what\'s coming:\\n- New pages in the `Consensus` section:\\n - A new `Node` page which provides a detailed breakdown of the counts and versions of all the nodes on the network.\\n - A new `Validator` page which provides a detailed view of a validator\'s performance over time, like when their attestation was seen by the Xatu nodes.\\n- A new `Data Availability` section including analysis of:\\n - A \\"Custody\\" deepdive which explores the data availability of our peers on the network (in collaboration with the EF P2P Team)\\n- A new `Execution Layer` section including analysis of:\\n - State size\\n - OPCODE and precompile usage\\n - Various state explorations, like size growth and hypothetical stateless explorations (in collaboration with the [EF Stateless Consensus Team](https://x.com/StatelessEth))\\n - Client execution time - how long it takes for all E
1xecution Layer clients to _execute_ a block\\n- A new `Fork Viewer` section which follows in the footsteps of [Forky](https://github.com/ethpandaops/forky) by providing a visual representation of the head of the chain _of our peers_.\\n\\n## A transformational backend\\nWe went back to the drawing board on a lot of the fundamental tooling in our existing data collection pipeline to help us achieve our goals. We ended up creating **8** new projects. Some of these projects are useful outside of the Ethereum ecosystem, and we\'re open sourcing them to the community.\\n\\n| Project | Description |\\n| --- | --- |\\n| [lab](https://github.com/ethpandaops/lab) | the new Lab frontend - built from the ground up. |\\n| [lab-backend](https://github.com/ethpandaops/lab-backend) | a new backend built from scratch to serve the Lab frontend. |\\n| [clickhouse-build-tool (CBT)](https://github.com/ethpandaops/cbt) | a simple, generic ClickHouse-focused data transformation tool that provides fast idempotent transformations with pure SQL or external scripts. Similar to DBT. |\\n| [cbt-api](https://github.com/ethpandaops/cbt-api) | an generic OpenAPI generator for CBT projects, providing both the spec & the server implementation to route queries to Clickhouse. |\\n| [xatu-cbt](https://github.com/ethpandaops/xatu-cbt) | contains the CBT models for the Xatu data. |\\n| [clickhouse-proto-gen](https://github.com/ethpandaops/clickhouse-proto-gen) | a tool for generating ClickHouse Protobuf schemas from Clickhouse tables. |\\n| [cartographoor](https://github.com/ethpandaops/cartographoor) | a single port-of-call for all our network specifications. |\\n| [xcli](https://github.com/ethpandaops/xcli) | a CLI for easily developing and testing new features in the Lab platform locally. |\\n\\nThese projects chain together to significantly simplify the process of building new features:\\n1. A new model is added to `xatu-cbt`\\n2. `clickhouse-proto-gen` generates the Protobuf schemas for the new model.\\n3. `cbt-api` generates the new OpenAPI spec, and the server implementation to route queries to Clickhouse.\\n4. `lab` uses the new OpenAPI spec to generate typescript types for interacting with the new model within the Lab frontend.\\n\\nThe entire pipeline runs automatically through code generation, requiring no manual configuration. :rocket:\\n\\n### The power of CBT <InlineLogo src=\\"/img/cbt.png\\" alt=\\"CBT\\" />\\n\\n<div style={{ display: \'flex\', flexWrap: \'wrap\', alignItems: \'flex-start\', gap: \'20px\', marginTop: \'20px\' }}>\\n <div style={{ flex: \'1 1 300px\', minWidth: \'0\' }}>\\n The largest breakthrough in this new architecture is [CBT](https://github.com/ethpandaops/cbt) <InlineLogo src=\\"/img/cbt.png\\" alt=\\"CBT\\" />. CBT lets us easily transform and compose data into interconnected tables, aggregating and consolidating it into new formats that build upon each other. The Lab is the most direct beneficiary of these new transformation tables, but the tables are also useful for other projects, both within ethPandaOps and the community.\\n\\n This abstraction simplifies complexity while compounding the value of our data.\\n </div>\\n\\n <img\\n src=\\"/img/blog/the-new-lab/cbt-trojan-horse.png\\"\\n alt=\\"CBT Trojan Horse\\"\\n style={{\\n maxWidth: \'100%\',\\n height: \'auto\',\\n width: \'350px\',\\n flex: \'0 0 auto\'\\n }}\\n />\\n</div>\\n\\n[CBT has a web interface](https://cbt.mainnet.ethpandaops.io\\") which provides a visual representation of the data transformation pipeline in real time. For example, the `fct_block` table is built on top of other CBT<InlineLogo src=\\"/img/cbt.png\\" alt=\\"CBT\\" /> tables in the form of `int_block_canonical` and `fct_block_head`. We\'ll be making a blog post about the inner workings of CBT soon. \\n\\n<ClickableImage\\n src=\\"/img/blog/the-new-lab/cbt-mainnet.png\\"\\n alt=\\"CBT Mainnet showing the dependencies of the fct_block table\\"\\n caption=\\"CBT Mainnet showing the dependencies of the fct_block table\\"\\n/>\\n\\n#### New Clickhouse tables\\nWe\'ll be publishing over 40 new transformation tables to the [Xatu Dataset](/data/xatu) <InlineLogo src=\\"/img/xatu.png\\" alt=\\"Xatu\\" /> in the coming weeks. These tables directly power The Lab, meaning **everything you see in The Lab will be directly available to the community via the [Xatu Dataset](/data/xatu) <InlineLogo src=\\"/img/xatu.png\\" alt=\\"Xatu\\" />.** These tables are nearly always faster to query than the existing raw tables that we already publish, so if you want to build something new on top of the data we publish, these tables will be a great place to start.\\n\\n<detail
1s>\\n<summary>View all new tables</summary>\\n\\n- fct_attestation_correctness_by_validator_canonical\\n- fct_attestation_correctness_by_validator_head\\n- fct_attestation_correctness_canonical\\n- fct_attestation_correctness_head\\n- fct_attestation_first_seen_chunked_50ms\\n- fct_attestation_liveness_by_entity_head\\n- fct_attestation_observation_by_node\\n- fct_block\\n- fct_block_blob_count\\n- fct_block_blob_count_head\\n- fct_block_blob_first_seen_by_node\\n- fct_block_data_column_sidecar_first_seen_by_node\\n- fct_block_first_seen_by_node\\n- fct_block_head\\n- fct_block_mev\\n- fct_block_mev_head\\n- fct_block_proposer\\n- fct_block_proposer_entity\\n- fct_block_proposer_head\\n- fct_data_column_availability_by_epoch\\n- fct_data_column_availability_by_slot\\n- fct_data_column_availability_by_slot_blob\\n- fct_head_first_seen_by_node\\n- fct_mev_bid_count_by_builder\\n- fct_mev_bid_count_by_relay\\n- fct_mev_bid_highest_value_by_builder_chunked_50ms\\n- fct_prepared_block\\n- int_attestation_attested_canonical\\n- int_attestation_attested_head\\n- int_attestation_first_seen\\n- int_beacon_committee_head\\n- int_block_blob_count_canonical\\n- int_block_canonical\\n- int_block_mev_canonical\\n- int_address_first_access\\n- int_address_last_access\\n- int_address_storage_slot_first_access\\n- int_address_storage_slot_last_access\\n\\n\\n</details>\\n\\n## Wrapping up\\nWe\'re extremely excited to be launching The Lab and we hope you enjoy using it. We\'d love to hear your feedback or suggestions, please reach out to us on [Twitter](https://x.com/ethpandaops) or join the [Xatu Telegram Group](https://t.me/+JanoQFu_nO8yNzQ1). \\n\\n<div style={{ textAlign: \'center\', marginTop: \'40px\', fontSize: \'1.5rem\', fontWeight: \'bold\' }}>\\n \\nSee you soon for Fusaka!\\n\\n\ud83d\ude80\ud83d\ude80\ud83d\ude80\\n</div>"},{"id":"fusaka-bandwidth-estimation","metadata":{"permalink":"/posts/fusaka-bandwidth-estimation","source":"@site/blog/2025-10-03-fusaka-bandwidth-estimations/index.md","title":"Fusaka bandwidth estimation","description":"Analysis of bandwidth used by various types of nodes after Fusaka","date":"2025-10-03T00:00:00.000Z","tags":[{"inline":true,"label":"fusaka","permalink":"/posts/tags/fusaka"}],"readingTime":12.27,"hasTruncateMarker":false,"authors":[{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"fusaka-bandwidth-estimation","title":"Fusaka bandwidth estimation","authors":["parithosh","samcm"],"description":"Analysis of bandwidth used by various types of nodes after Fusaka","tags":["fusaka"],"image":"/img/blog/fusaka-bandwidth-needs/cyberpunk-blob-estimation.png","relatedLinks":[{"name":"Manu\'s PeerDAS Deep Dive","url":"https://hackmd.io/@manunalepa/r1H8CtZlR"}]},"unlisted":false,"prevItem":{"title":"Introducing The Lab","permalink":"/posts/the-lab"},"nextItem":{"title":"Fusaka-Devnet-5 BPO Analysis","permalink":"/posts/fusaka-devnet-5-bpo-analysis"}},"content":"## Introduction\\n[Fusaka has just launched on the Holesky testnet](https://x.com/ethPandaOps/status/1973325647020957804), and it\'s quickly approaching Mainnet. The fork includes a new feature called PeerDAS which is a new way to distribute blobs to nodes on the network. In PeerDAS, nodes will be responsible for \\"custodying\\" different pieces of the blob data determined by the amount of ETH they\'re validating with. The idea is pretty simple: if you have a lot of ETH staked you probably should have enough resources to handle more blobs, and vice versa. This preserves decentralization ensuring to not exclude home stakers from the network while still providing the benefits of a larger blob count.\\n\\n**This post will focus on the bandwidth and disk usage requirements for various types of nodes on the network after Fusaka.**\\n\\nWe\'ll approach this analysis from two different angles:\\n- The theoretical requirements for nodes on the network\\n- What we observed from testing on devnets\\
1n\\n## PeerDAS\\n\\nPeerDAS divides our existing 128kB blobs into 128 columns of ~2KB each. These columns are then broadcast individually on the peer-to-peer layer, and nodes custody different subsets of these columns based on the amount of ETH they\'re validating with. [Read more about PeerDAS on ethereum.org](https://ethereum.org/roadmap/fusaka/peerdas/).\\n\\nWith the addition of erasure coding (2x blob size), this enables a theoretical 8x scaling limit compared to the current \\"everyone downloads everything\\" model. Nodes can verify data availability through sampling without storing complete blobs. This introduces new types of nodes that are possible on the network with different requirements for each.\\n\\n**Three node types emerge**:\\n- **Supernode/Backbone node**: Validators with combined balance \u22654,096 ETH must subscribe to all 128 column subnets and custody all data for 4096 epochs (~18 days). These nodes are basically forming the backbone of the network. They will have a higher processing and p2p load than the other nodes on the network. However, the stake they represent is non-trivial and these are likely sophisticated entities.\\n- **Validating node**: Regular validators (with 1 validator) subscribe to their assigned 8+ random column subnets. Therefore, they would process 1/8th (or more) of the data a supernode would, implying lower p2p,disk and CPU load. This is the benefit that the average home staker would receive.\\n- **Full node**: Non-validating nodes participating in 4 column subnets. They would process exactly 1/32th the data a supernode would, implying significantly lower p2p, disk and CPU load. However they still contribute to data availability even without the staking requirement. This is mostly a benefit for the average RPC node that is primarily interested in the state. *(Note: If you intend to provide blobs over the Beacon API you should be a supernode as you require all the columns to reconstruct the blob.)*\\n\\n### BPOs\\nFusaka also introduces the concept of a Blob Parameter Only fork via [EIP-7892](https://eips.ethereum.org/EIPS/eip-7892). These are forks that can only modify the blob count values without the coordination overhead of a full hard fork. Core devs are currently planning on shipping Fusaka with BPO1 and BPO2 to monitor the effects on Mainnet, and will then proceed with more aggressive scaling for BPO3, BPO4, etc, once data has been collected and analysed.\\n\\nBased on [testing and data collected over devnets](/posts/fusaka-devnet-5-bpo-analysis/) we arrived at the following proposal for a blob schedule for mainnet:\\n\\nexport const BPOScheduleTable = () => {\\n const schedule = [\\n { phase: \\"Fusaka\\", target: 6, max: 9 },\\n { phase: \\"BPO1\\", target: 10, max: 15 },\\n { phase: \\"BPO2\\", target: 14, max: 21 }\\n ];\\n\\n return (\\n <div style={{\\n width: \'100%\',\\n margin: \'20px 0\',\\n borderRadius: \'6px\',\\n overflow: \'hidden\',\\n border: \'1px solid var(--ifm-color-emphasis-200)\'\\n }}>\\n <div style={{\\n display: \'flex\',\\n backgroundColor: \'var(--ifm-color-primary)\',\\n color: \'white\',\\n padding: \'12px 16px\',\\n fontWeight: 600\\n }}>\\n <div style={{ flex: \'0 0 33%\' }}>Phase</div>\\n <div style={{ flex: \'0 0 33%\', textAlign: \'center\' }}>Target Blobs</div>\\n <div style={{ flex: \'0 0 33%\', textAlign: \'center\' }}>Max Blobs</div>\\n </div>\\n {schedule.map((item, index) => (\\n <div\\n key={index}\\n style={{\\n display: \'flex\',\\n alignItems: \'center\',\\n backgroundColor: index % 2 === 0 ? \'var(--ifm-background-surface-color)\' : \'var(--ifm-color-emphasis-100)\',\\n padding: \'10px 16px\',\\n borderBottom: index < schedule.length - 1 ? \'1px solid var(--ifm-color-emphasis-200)\' : \'none\',\\n fontSize: \'0.95rem\'\\n }}\\n >\\n <div style={{ flex: \'0 0 33%\', fontWeight: \'bold\' }}>{item.phase}</div>\\n <div style={{ flex: \'0 0 33%\', textAlign: \'center\', fontWeight: \'bold\', color: \'var(--ifm-color-primary)\' }}>{item.target}</div>\\n <div style={{ flex: \'0 0 33%\', textAlign: \'center\', fontWeight: \'bold\', color: \'var(--ifm-color-primary)\' }}>{item.max}</div>\\n </div>\\n ))}\\n </div>\\n );\\n};\\n\\n<BPOScheduleTable />
1\\n\\n## Theoretical Requirements\\nThe Ethereum network is a complex and evolving system, and its very difficult to nail down the exact requirements for the network after Fusaka given the amount of moving parts. However, we can make some educated guesses based on the current state of the network and the proposed changes.\\n\\nTo help us understand the requirements for the network after Fusaka we\'ll start by understanding the current state of the network. Mainnet is currently running with `6 Target 9 Max` blobs. The `Target` blob count is the one thats important here as it\'s the equilibrium point of the blob fee market and we can roughly expect blob count usage to trend there over time.\\n\\n> Reminder: Pre-Fusaka Blobs are 128kB. Post-Fusaka Blobs are 256kB (128kB for the blob, 128kB for the erasure coding.)\\n\\nTo help, we\'ve created a resource calculator that can help identify the requirements for different node types and blob counts. **This data needs to be taken with a grain of salt as it is theoretical and assumes a few things.** Make sure you change the node type to see the different requirements, and to check the notes for each metric below.\\n\\nimport FusakaResourceCalculator from \'@site/src/components/FusakaResourceCalculator\';\\n\\n<FusakaResourceCalculator />
1\\n\\nThose who flicked through the calculator will have noticed that there is a significant increase in the requirements for local block building. This is a result of the new erasure coding. Given [EIP-7870](https://eips.ethereum.org/EIPS/eip-7870) the maximum upload bandwidth for a local block builder is 50mb, but our theoretical calculator shows we surpass that with the BPO1 blob count value of 10! What gives? Luckily core devs have a few tricks up their sleeves to help with this:\\n\\n#### 1. Distributed Blob Building\\nWhen a block is built locally the proposer publishes the block and then all 128 columns. When another node receives the block it will first check it\'s own mempool for the blobs referenced in the block. If it finds them it is able to skip the download and use the local blobs instead. The main benefit here is that the time for a block to be *attestable* is reduced for the validator as it doesn\'t have to wait for the blobs to be received via gossip. **This has a direct benefit to the block proposer as their proposed blocks will be attestable quicker.** There is also the nice side effect of the receiving node potentially being able to avoid downloading the blobs via gossipsub.\\n\\n#### 2. Using a MEV-Relay\\n[With 86% of blocks being provided by MEV Relays over the last 14 days](https://mevboost.pics/), the reality is that a significant portion of the network will have the help of a MEV Relay to publish their blocks. While we don\'t want to rely on these centralized entities, they are a fact of mainnet, and combined with the other two tricks above, we have confidence that the network will handle the load.\\n\\n#### 3. `--max-blobs` flag\\nConsensus Layer clients may ship with a `--max-blobs` flag that will limit the number of blobs that the client will propose blocks with. Theres no point in building a block with more blobs than you can upload, so this will be a good way to ensure you don\'t miss any proposals. This feature is still a WIP and may not be available in your client as soon as you update to a mainnet compatible release.\\n\\n## Devnet Data\\nTaking a step back from the theoretical requirements, we can look at the data collected from our devnets to get a better sense of the requirements for the network.\\n\\n- Multiple devnets were run over the last few months, but we will focus on data from fusaka-devnet-5 due to its size.\\n- ~1700 node devnet (1/8th mainnet) that was deployed in collaboration with Sunnyside labs.\\n- 5 BPOs were configured that tested blob counts as high as 72 blobs. We will focus on the range below 21 blobs as those are the suggested max values for BPO1 and BPO2.\\n- We test against our existing node types (supernodes, validating nodes, full nodes) as each has differing requirements.\\n\\n### Nodes used for testing\\nOur infrastructure looked as follows:\\n- For Supernodes:\\n - 8 vCPU/32GB RAM/640GB SSD\\n - 1000 Mbps upload and download speeds\\n- For Validating nodes:\\n - 8 vCPUs / 16GB RAM\\n - 100/50 Mbps Download/Upload limited using traffic control\\n - Some ARM nodes were also run with varying number of validator counts\\n- For Full nodes:\\n - 4 vCPUs / 8GB RAM \\n - 50/15Mbps Download/Upload limited using traffic control\\n - Run by the Sunnyside Labs team\\n\\nDuring the test phase, we always had 2 phases per BPO. Half the duration was spent using MEV and the other half utilizing local building alone. This allowed us to collect datapoints for both types of network setups. \\n\\n### Devnet observations for BPO1\\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/10blobs-supernodes.png\\"\\nalt=\\"Bandwidth used for 10 blobs in a supernode\\"\\ncaption=\\"Figure 1.1: Bandwidth used for 10 blobs in a supernode\\"\\n/>\\n\\nSupernodes show significant bandwidth spikes with 10 blobs: receive peaks at ~200 Mb/s, transmit peaks at ~250 Mb/s, with baselines around 50-100 Mb/s. \\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/10blobs-validatingnodes.png\\"\\nalt=\\"Bandwidth used for 10 blobs in a validating node/homestaker\\"\\ncaption=\\"Figure 1.2: Bandwidth used for 10 blobs in a validating node\\"\\n/>\\n\\nValidating nodes (home stakers) show dramatically reduced requirements: ~15-20 Mb/s receive, ~10-15 Mb/s transmit. This demonstrates PeerDAS\'s key benefit to home stakers.\\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/10blobs-fullnodes.png\\"\\nalt=\\"Bandwidth used for 10 blobs in a full node\\"\\ncaption=\\"Figure 1.3: Bandwidth used for 10 blobs in a fullnodes\\"\\n/>\\n\\nFull nodes have minimal bandwidth needs: ~4-8 Mb/s for both receive and transmit.\\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/10blobs-getblobsrate.png\\"\\nalt=\\"% of blobs fetched from EL instead of via gossip\\"\\ncaption=\\"Figure 1.4: % of blobs fetched from EL instead of via gossip\\"\\n/>\\n\\nThe data shows significant variation in blob retrieval strategies across the network, with fetch rates from the Execution Layer ranging from 40-100%. This implies that a large section of blobs can be fetched before they are needed in the block validation pipeline, reducing the bandwidth needs at the critical portion of the slot. This data however is only from the devnet and is not proof that we\'d see similar numbers on Mainnet.\\n\\n**Summary for BPO1**: The observed bandwidth requirements for validating nodes (home stakers) at 10 blobs are well within [EIP-7870](https://eips.ethereum.org/EIPS/eip-7870) specifications, which require 50 Mbps download and 25 Mbps upload. Our devnet data shows peaks of ~15-20 Mb/s in both directions, leaving headroom for other operations.\\n\\n### Devnet observations for BPO2\\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/14blobs-supernodes.png\\"\\nalt=\\"Bandwidth used for 14 blobs in a supernode\\"\\ncaption=\\"Figure 2.1: Bandwidth used for 14 blobs in a supernode\\"\\n/>\\n\\nWith 14 blobs, supernodes face increased demands: receive spikes to ~400 Mb/s, transmit to ~300 Mb/s. However, ignoring the spikes, the baselines data usage seems to be around 100 Mb/s. The higher blob count clearly amplifies bandwidth requirements for supernodes.\\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/14blobs-validatingnodes.png\\"\\nalt=\\"Bandwidth used for 14 blobs in a validating node/homestaker\\"\\ncaption=\\"Figure 2.2: Bandwidth used for 14 blobs in a validating node\\"\\n/>\\n\\nValidating nodes maintain reasonable requirements even at 14 blobs: ~17-25 Mb/s for both directions. \\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/14blobs-fullnodes.png\\"\\nalt=\\"Bandwidth used for 14 blobs in a full node\\"\\ncaption=\\"Figure 2.3: Bandwidth used for 14 blobs in a fullnodes\\"\\n/>\\n\\nFull nodes continue to show minimal bandwidth requirements at 14 blobs: ~8 Mb/s for both receive and transmit. \\n\\n<ClickableImage\\nsrc=\\"/img/blog/fusaka-bandwidth-needs/14blob-getblobsrate.png\\"\\nalt=\\"% of blobs fetched from EL instead of via gossip\\"\\ncaption=\\"Figure 2.4: % of blobs fetched from EL instead of via gossip\\"\\n/>\\n\\nAt 14 blobs, the network continues to show varied blob retrieval patterns, with EL fetch rates ranging from 20-90%. We see that the hit rate reduces with an increasing number of blobs even for controlled networks. This data however is not representative of Mainnet behaviour and we\'d need additional monitoring to understand the behaviour on Mainnet. \\n\\n\\
1n**Summary for BPO2**: Even at the higher blob count of 14, validating nodes remain within [EIP-7870](https://eips.ethereum.org/EIPS/eip-7870) bandwidth limits for home stakers (50 Mbps download, 25 Mbps upload). The observed ~17-25 Mb/s in both directions still is achievable for the average home staker with local proposals.\\n\\n## General Caveats\\n- The BPO traffic is hard to nail down because it is unclear how much the public mempool traffic will account for with increased blobs. Theoretically, the mempool should see more traffic as we increase the target blob count - but this is only an assumption due to the existence of private mempools.\\n- The values provided in the document can be used as high level guidelines, please ensure that you have additional bandwidth available to account for engineering margins. Once Fusaka is live on Mainnet, we would obtain live data and can update the analysis with more accurate information.\\n- Theoretical values proposed earlier in the document do not capture the nuances of running a node such as mempool traffic on the EL, providing sync state to peers and other background gossip. \\n\\n## Summary\\n- **Home stakers**: Validating nodes require ~25 Mb/s bandwidth (within margins of [EIP-7870](https://eips.ethereum.org/EIPS/eip-7870) limits), allowing Mainnet to scale to 14 blobs safely.\\n- **Effective load distribution**: Supernodes handle backbone requirements (200-400 Mb/s peaks) while full nodes operate minimally (4-8 Mb/s), demonstrating PeerDAS\'s scalability benefits.\\n- **Distributed blob building helps**: Local blob retrieval from execution layer mempools can help reduce gossip bandwidth during critical validation periods.\\n\\nThis data supports proceeding with Fusaka\'s rollout schedule while maintaining decentralization. We will continue to monitor the Fusaka rollout on Mainnet as conditions may differ from devnet testing.\\n\\n## Follow up reading:\\nPlease also read the analysis provided by the [Sunnyside labs](https://x.com/testinprod_io) team, found [here](https://testinprod.notion.site/Draft-Sunnyside-Devnet-Updates-09-29-26f8fc57f546808cbc83c20ca7c386f8). They go into significantly more detail about the breakdown of types of traffic as well as optimisations."},{"id":"fusaka-devnet-5-bpo-analysis","metadata":{"permalink":"/posts/fusaka-devnet-5-bpo-analysis","source":"@site/blog/2025-09-18-fusaka-devnet-5-bpo-analysis/index.md","title":"Fusaka-Devnet-5 BPO Analysis","description":"Analysis of blob parameter optimization values from fusaka-devnet-5 to inform safe initial values for the Fusaka fork.","date":"2025-09-18T00:00:00.000Z","tags":[{"inline":true,"label":"fusaka","permalink":"/posts/tags/fusaka"}],"readingTime":12.2,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"fusaka-devnet-5-bpo-analysis","title":"Fusaka-Devnet-5 BPO Analysis","authors":["samcm"],"description":"Analysis of blob parameter optimization values from fusaka-devnet-5 to inform safe initial values for the Fusaka fork.","tags":["fusaka"],"image":"/img/blog/fusaka-devnet-5-bpo-analysis.jpg","relatedLinks":[{"name":"Fusaka-Devnet-5","url":"/networks/fusaka-devnet-5"},{"name":"Xatu Analysis","url":"https://github.com/ethpandaops/xatu-analysis"},{"name":"Fusaka Scheduling","url":"https://notes.ethereum.org/@jtraglia/fusaka_scheduling"}]},"unlisted":false,"prevItem":{"title":"Fusaka bandwidth estimation","permalink":"/posts/fusaka-bandwidth-estimation"},"nextItem":{"title":"Scaling Ethereum: The Path to 45M Gas Limit and Beyond","permalink":"/posts/gaslimit-scaling"}},"content":"export const BlobViabilityRow = ({ items }) => {\\n const getStatusColor = (status) => {\\n switch(status) {\\n case \'viable\': return \'#28a745\';\\n case \'marginal\': return \'#ffc107\';\\n case \'not-viable\': return \'#dc3545\';\\n default: return \'#6c757d\';\\n }\\n };\\n\\n const getStatusIcon = (status) => {\\n switch(status) {\\n case \'viable\': return \'\u2713\';\\n case \'marginal\': return \'\u26a0\';\\n case \'not-viable\': return \'\u2717\';\\n default: return \'?\';\\n }\\n };\\n\\n const getStatusText = (status) => {\\n switch(status) {\\n case \'viable\': return \'Viable\';\\n case \'marginal\': return \'Marginal\';\\n case \'not-viable\': return \'Not Viable\';\\n default: return \'Unknown\';\\n }\\n };\\n\\n return (\\n <div style={{\\n display: \'grid\',\\n gridTemplateColumns: \'repeat(3, 1fr)\',\\n gap: \'12px\',\\n margin: \'20p
1x 0\'\\n }}>\\n {items.map((item, index) => (\\n <div key={index} style={{\\n display: \'flex\',\\n alignItems: \'center\',\\n gap: \'10px\',\\n padding: \'12px\',\\n backgroundColor: \'var(--ifm-color-emphasis-100)\',\\n borderLeft: `4px solid ${getStatusColor(item.status)}`,\\n borderRadius: \'4px\',\\n minHeight: \'60px\'\\n }}>\\n <div style={{\\n width: \'28px\',\\n height: \'28px\',\\n borderRadius: \'50%\',\\n backgroundColor: getStatusColor(item.status),\\n color: \'white\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n fontSize: \'16px\',\\n fontWeight: \'bold\',\\n flexShrink: 0\\n }}>\\n {getStatusIcon(item.status)}\\n </div>\\n <div style={{ flex: 1 }}>\\n <div style={{\\n fontWeight: \'bold\',\\n fontSize: \'0.95rem\',\\n color: getStatusColor(item.status),\\n marginBottom: \'2px\'\\n }}>\\n {item.count} Blobs\\n </div>\\n <div style={{\\n fontSize: \'0.85rem\',\\n color: \'var(--ifm-color-emphasis-600)\',\\n lineHeight: 1.2\\n }}>\\n {getStatusText(item.status)}\\n </div>\\n </div>\\n </div>\\n ))}\\n </div>\\n );\\n};\\n\\n## Intro\\n\\nFusaka is quickly approaching, and it\'s time to start picking **B**lob **P**arameter **O**nly fork values. These are mini hard forks where the only thing that changes is the blob count. They provide a mechanism for core devs to safely scale the chain without the coordination overhead of a fully-fledged hard fork. The current idea amongst devs is to ship Fusaka with safe initial blob count values, collect data, then more aggressively scale up with more confidence shortly after.\\n\\nBut what do we set those initial values to? We need to pick values that will be safe while also providing enough data to provide insight in to future BPO values. Enter `fusaka-devnet-5`.\\n\\n## Fusaka-Devnet-5\\n> In collaboration with [Sunnyside Labs](https://x.com/testinprod_io) \u2764\ufe0f\\n\\nThis network is more of a smoke test than anything. We\u2019re checking things work on a fundamental level but not much beyond that.\\n\\nThe network has a high node count (1-2k) to help bridge this gap, but this is still ~1/8th of Mainnet. Unfortunately this brings a certain amount of chaos. Individually tending to nodes becomes unviable, and the holes in the ship quickly become insurmountable for humans to plug. As of time-of-writing there is between 50-100 nodes that are failing to attest correctly above a 50% rate.\\n\\nThroughout the lifecycle of this network we\'ve also stepped on a few rakes:\\n- **Node misconfigurations**\\n\\n Due to a misconfiguration in the Ansible on the ethPandaOps side, [~700 \\"full\\" nodes were custodying all columns while being bandwidth limited](https://discord.com/channels/595666850260713488/892088344438255616/1417906013511876761). This was fixed around 6PM UTC 17/09/25.\\n- **Multiple client bugs**\\n\\n This is a great outcome - it\'s always better to find these in the devnet cycle, but it provides another layer of noise to the data that is challenging to cut through.\\n\\n**Given the above there is only a relatively small window of usable data (~12hrs).** This is additionally challenging as some of the testing scenarios are relatively low volume. Given this landscape, we have fairly low confidence in the data found here, but it should do the job for unblocking us on initial testnet BPO values. **We will likely run a `fusaka-devnet-6` in the coming weeks to improve confidence in the numbers found here.**\\n\\n<center>**This data and analysis is merely a starting point.** </center>\\n\\n## Analysis\\nWe\'ll focus on the viability of these Max Blob Count options:\\n<BlobViabilityRow\\n items={[\\n { count: 15, status: \\"pending\\" },\\n { count: 21, status: \\"pending\\" },\\n { count: 32, status: \\"pending\\" }\\n ]}\\n/>\\n\\nWe utilize a metric called `Head attestation immediately correct %`. We define this as an attestation that voted for the block in their duty slot (*not* a parent block). This means that the validator saw the block and the required columns before 4s. If an attester votes for a parent block it means they didn\'t see the block/columns in time. This bundles up a lot of the complexity within the network, like distributed blob building, propagation, block processing time, etc, into a
1nice single metric.\\n\\nWith Xatu we have the ability to filter and group this data on both the proposer side AND the attester side on multiple dimensions. E.g. \\"Show slots that were proposed by all supernodes and attested to by Lodestar regular nodes when proposed via mev-relay, grouped by proposer CL type\\"\\n\\n### Test 1: Can a full node propose a block without MEV-Boost?\\nAs per [EIP-7870](https://eips.ethereum.org/EIPS/eip-7870), full nodes have 50mb of upload bandwidth to play with. They have to gossip all 128 columns to their peers, and 66% of the slots attesters have to see those columns before 4s. This is a tough ask for our most bandwidth sensitive network contributors as the blob count scales, and we\'d expect to hit a tipping point somewhere here.\\n\\n#### Data Filter\\n- 12 hours of data\\n- **Only locally built blocks**\\n- **Proposed by full nodes**\\n - Excluding Prysm as it had a block proposal bug throughout the time window\\n - Excluding Nimbus as it is a severe outlier and indicates client specific issues\\n- Attested to by all node types (full + super)\\n - Excluding Nimbus (same reason as above)\\n - Excluding Lodestar as it has a sizable amount of nodes with issues throughout the time window\\n\\nThis network only has 10% of the effective balance running on full nodes. Combined with needing to filter out Prysm & Nimbus proposals, which are 40% of our full node effective balance, we\'re really struggling for data. Nonetheless we can still dive in.\\n\\n#### Results\\nGiven the above, we only have 100 slots of data to work with. We\'re really only checking if these blob counts are _fundamentally_ possible though, which helps.\\n\\n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-1-proposers.jpg\\"\\n alt=\\"Head Correctness grouped by proposer CL\\"\\n caption=\\"Figure 2.1: Head Correctness grouped by proposer CL\\"\\n/>\\n\\nFocusing on the proposer side here, there appears to be a fall off past 21 blobs. Lighthouse is a strong performer.\\n\\n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-1-attesters.jpg\\"\\n alt=\\"Head Correctness grouped by attester CL-EL\\"\\n caption=\\"Figure 2.2: Head Correctness grouped by attester CL-EL\\"\\n/>\\n\\nLooking at the attester side of the equation here. This is observed across all attester CL-EL combos which indicates that the proposers are under stress.\\n\\n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-1-lighthouse-attesters.jpg\\"\\n alt=\\"Head Correctness grouped by attester CL-EL\\"\\n caption=\\"Figure 2.3: Head Correctness grouped by attester CL-EL for blocks proposed by only Lighthouse\\"\\n/>\\n\\nWe noticed strong proposed performance with Lighthouse earlier so we can zoom in on just their block proposals. Surprisingly, the 21-31 blob bucket is showing signs of life and is theoretically possible. This indicates that there may be more juice to squeeze out of clients when it comes to pushing the blob count for our bandwidth sensitive operators. Maybe theres something lurking in libp2p implementations? One for the philosophers.\\n\\n#### Outcome\\n\\n<BlobViabilityRow\\n items={[\\n { count: 15, status: \\"viable\\" },\\n { count: 21, status: \\"viable\\" },\\n { count: 32, status: \\"marginal\\" }\\n ]}\\n/>\\n\\n### Test 2: Can supernodes see 128 columns at max blob count on time?\\nNodes will only attest to a block once they see all the columns they\'re custodying. In the case of Supernodes, this is all 128 columns. We need to assert that these nodes can consistently see all columns well-before 4s. The worry is that it only takes 1 column to arrive late and a large % of validators won\'t attest to the block.\\n\\n#### Data Filter:\\n- ~14 hours of data\\n- All block sources (local & mev relay)\\n- Proposed by supernodes\\n - Excluding Prysm as it had a block proposal bug throughout the time window\\n - Excluding Nimbus as it is a severe outlier and indicates client specific issues\\n- **Attested to by supernodes**\\n - Excluding Nimbus (same reason as above)\\n - Excluding Lodestar as it has a sizable amount of nodes with issues throughout the time window\\n\\n#### Results\\n\\
1n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-2-proposers.jpg\\"\\n alt=\\"Head Correctness grouped by proposer CL\\"\\n caption=\\"Figure 2.1: Head Correctness grouped by proposer CL\\"\\n/>\\n\\n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-2-attesters.jpg\\"\\n alt=\\"Head Correctness grouped by attester CL+EL\\"\\n caption=\\"Figure 2.2: Head Correctness grouped by attester CL+EL\\"\\n/>\\n\\nBoth sides of the equation here show strong results. We don\'t appear to be hitting any bottlenecks in this configuration as blob count increases.\\n\\n<BlobViabilityRow\\n items={[\\n { count: 15, status: \\"viable\\" },\\n { count: 21, status: \\"viable\\" },\\n { count: 32, status: \\"viable\\" }\\n ]}\\n/>\\n\\n### Test 3: Can the network recover columns in time?\\nIf the proposer fails to publish a single column, can the network recover that column in time to save the slot? Some Lighthouse nodes have been configured to randomly withhold a column when they propose blocks.\\n\\n#### Results\\n\\nexport const RecoveryTable = () => {\\n const slots = [\\n { slot: 54989, correctness: 87.4, blobs: 32 },\\n { slot: 54636, correctness: 88.8, blobs: 27 },\\n { slot: 54606, correctness: 89.6, blobs: 30 },\\n { slot: 54096, correctness: 88.5, blobs: 26 },\\n { slot: 53689, correctness: 89.0, blobs: 27 }\\n ];\\n\\n return (\\n <div style={{\\n width: \'100%\',\\n margin: \'20px 0\',\\n borderRadius: \'6px\',\\n overflow: \'hidden\',\\n border: \'1px solid var(--ifm-color-emphasis-200)\'\\n }}>\\n <div style={{\\n display: \'flex\',\\n backgroundColor: \'var(--ifm-color-primary)\',\\n color: \'white\',\\n padding: \'12px 16px\',\\n fontWeight: 600\\n }}>\\n <div style={{ flex: \'0 0 25%\' }}>Slot</div>\\n <div style={{ flex: \'0 0 30%\', textAlign: \'center\' }}>Head Correctness %</div>\\n <div style={{ flex: \'0 0 20%\', textAlign: \'center\' }}>Blob Count</div>\\n <div style={{ flex: \'0 0 25%\', textAlign: \'center\' }}>Status</div>\\n </div>\\n {slots.map((item, index) => (\\n <div\\n key={index}\\n style={{\\n display: \'flex\',\\n alignItems: \'center\',\\n backgroundColor: index % 2 === 0 ? \'var(--ifm-background-surface-color)\' : \'var(--ifm-color-emphasis-100)\',\\n padding: \'10px 16px\',\\n borderBottom: index < slots.length - 1 ? \'1px solid var(--ifm-color-emphasis-200)\' : \'none\',\\n fontSize: \'0.95rem\'\\n }}\\n >\\n <div style={{ flex: \'0 0 25%\' }}>\\n <a href={`https://dora.fusaka-devnet-5.ethpandaops.io/slot/${item.slot}`} target=\\"_blank\\">\\n {item.slot}\\n </a>\\n </div>\\n <div style={{ flex: \'0 0 30%\', textAlign: \'center\' }}>{item.correctness}%</div>\\n <div style={{ flex: \'0 0 20%\', textAlign: \'center\' }}>{item.blobs}</div>\\n <div style={{ flex: \'0 0 25%\', textAlign: \'center\' }}>\\n <span style={{ color: \'#28a745\' }}>\u2713 Recovered</span>\\n </div>\\n </div>\\n ))}\\n </div>\\n );\\n};\\n\\n<RecoveryTable />\\n\\nAll tested slots successfully recovered the withheld columns and became canonical, achieving *immediate attestation correctness* that is around par for the network.\\n<BlobViabilityRow\\n items={[\\n { count: 15, status: \\"viable\\" },\\n { count: 21, status: \\"viable\\" },\\n { count: 32, status: \\"viable\\" }\\n ]}\\n/>\\n\\n### Test 4: Can new nodes join the network?\\n[Syncoor](https://syncoor.fusaka-devnet-5.ethpandaops.io/#/?directory=Checkpoint+Sync+%2B+CL+Supernode&network=fusaka-devnet-5) has been deployed to fusaka-devnet-5, and has been doing frequent sync tests. Shoutout to [Skylenet](https://x.com/skylenet) on our team for this shiny new tool! The Syncoor UI has in depth metrics around runtime and utilization.\\n\\n#### Results\\nNew nodes are able to join the network in a pretty linear time fashion across both full nodes and supernodes. Syncoor conducted ~3500 sync tests over the duration of the network \ud83d\udc51\\n\\n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-4-overview-full.jpg\\"\\n alt=\\"Full nodes sync to head\\"\\n caption=\\"Figure 4.1: Full nodes are able to sync to the head of the network\\"\\n/>\\n\\n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-4-overview-supernodes.jpg\\"\\n alt=\\"Super nodes sync to head\\"\\n caption=\\"Figure 4.2: Super nodes are able to sync to the head of the network\\"\\n/>\\n\\n\\n\\n<ClickableImage\\n src=\\"/img/blog/fusaka-devnet-5/test-4-erigon.jpg\\"\\n alt=\\"Erigon sync times on fusaka-devnet-5\\"\\n caption=\\"Figure 4.3: Erigon sync times on fusaka-devnet-5\\"\\n/>\\n\\nErigon is a bit of an outlier here. It\'s times are increasing at a substantial rate, and it is generally much slower than other ELs, but this is most likely related to the lack of torrent infrastru
1cture on the network. This is something we\'re working on and will be available in the near future.\\n\\n<BlobViabilityRow\\n items={[\\n { count: 15, status: \\"viable\\" },\\n { count: 21, status: \\"viable\\" },\\n { count: 32, status: \\"viable\\" }\\n ]}\\n/>\\n\\n## Conclusion\\n`fusaka-devnet-5` provided more challenges than we were hoping but still provided valuable insight in to how the network performs as the blob count increases. **It must be reiterated that the data collected here has a small sample size, and that `fusaka-devnet-5` does not represent Mainnet**.\\n\\nexport const ViableOptions = () => {\\n const options = [\\n {\\n max: 15,\\n title: \\"\\",\\n description: \\"A safe starting point\\",\\n benefits: [\\n \\"\u2713 Full node local block proposals work reliably\\",\\n \\"\u2713 Supernodes handle all 128 columns comfortably\\",\\n \\"\u2713 Network recovery mechanisms function smoothly\\",\\n \\"\u2713 New nodes can sync without issues\\"\\n ]\\n },\\n {\\n max: 21,\\n title: \\"\\",\\n description: \\"Better data for future increases\\",\\n benefits: [\\n \\"\u2713 Provides valuable data for future scaling\\",\\n \\"\u2713 40% more blob capacity than the 15 option\\",\\n \\"\u2713 Still within safe operational limits\\"\\n ]\\n },\\n {\\n max: 32,\\n title: \\"\\",\\n description: \\"Notable challenges \\",\\n benefits: [\\n \\"\u26a0 Full nodes struggle to publish blocks without the help of MEV Relays\\",\\n \\"\u2713 Supernodes handle the load adequately\\",\\n \\"\u2713 Network recovery still functional\\"\\n ],\\n }\\n ];\\n\\n return (\\n <div style={{\\n margin: \'40px 0\',\\n padding: \'30px\',\\n }}>\\n <div style={{\\n display: \'grid\',\\n gridTemplateColumns: \'repeat(auto-fit, minmax(280px, 1fr))\',\\n gap: \'20px\'\\n }}>\\n {options.map((option, index) => (\\n <div key={index} style={{\\n padding: \'24px\',\\n backgroundColor: option.notViable ? \'var(--ifm-color-danger-lightest)\' : \'var(--ifm-background-surface-color)\',\\n borderRadius: \'8px\',\\n border: `2px solid ${option.notViable ? \'var(--ifm-color-danger-light)\' : \'var(--ifm-color-primary-lighter)\'}`,\\n position: \'relative\',\\n opacity: option.notViable ? 0.9 : 1\\n }}>\\n <div style={{\\n position: \'absolute\',\\n top: \'-15px\',\\n left: \'20px\',\\n backgroundColor: option.notViable ? \'var(--ifm-color-danger)\' : \'var(--ifm-color-primary)\',\\n color: \'white\',\\n padding: \'6px 16px\',\\n borderRadius: \'20px\',\\n fontSize: \'1.2rem\',\\n fontWeight: \'bold\'\\n }}>\\n Max = {option.max}\\n </div>\\n {option.notViable && (\\n <div style={{\\n position: \'absolute\',\\n top: \'-15px\',\\n right: \'20px\',\\n backgroundColor: \'var(--ifm-color-danger)\',\\n color: \'white\',\\n padding: \'4px 12px\',\\n borderRadius: \'12px\',\\n fontSize: \'0.8rem\',\\n fontWeight: \'bold\',\\n textTransform: \'uppercase\'\\n }}>\\n Not Viable\\n </div>\\n )}\\n <h4 style={{\\n marginTop: \'20px\',\\n marginBottom: \'12px\',\\n color: \'var(--ifm-heading-color)\',\\n fontSize: \'1.1rem\'\\n }}>\\n {option.title}\\n </h4>\\n <p style={{\\n marginBottom: \'20px\',\\n fontSize: \'0.95rem\',\\n color: \'var(--ifm-color-emphasis-700)\',\\n lineHeight: 1.5\\n }}>\\n {option.description}\\n </p>\\n <div style={{\\n marginBottom: \'20px\',\\n padding: \'16px\',\\n backgroundColor: \'var(--ifm-color-emphasis-100)\',\\n borderRadius: \'6px\'\\n }}>\\n <h5 style={{\\n fontSize: \'0.9rem\',\\n marginBottom: \'12px\',\\n textTransform: \'uppercase\',\\n color: \'var(--ifm-color-emphasis-600)\',\\n fontWeight: 600\\n }}>\\n Test Results\\n </h5>\\n <div style={{ fontSize: \'0.9rem\' }}>\\n {option.benefits.map((benefit, idx) => (\\n <div key={idx} style={{\\n marginBottom: \'8px\',\\n color: benefit.startsWith(\'\u26a0\') ? \'#ff9800\' : \'#28a745\'\\n }}>\\n {benefit}\\n </div>\\n ))}\\n </div>\\n </div>\\n </div>\\n ))}\\n </div>\\n </div>\\n );\\n};\\n\\n<ViableOptions />\\n\\n### Potential Initial BPO Schedule\\nGiven the viability of the above, we propose a schedule that combines safety, throughput increases, and enough data to inform future BPO decisions. As we go through the testnet stage for Fusaka this may need to be adju
1sted as new data comes to light.\\n\\n| Phase | Target Blobs | Max Blobs | Change | Notes |\\n| --- | :---: | :---: | :---: | --- |\\n| **Electra** | 6 | 9 | | Live now |\\n| **Fusaka** | 6 | 9 | +0% | Initial PeerDAS activation |\\n| **BPO 1** | 10 | 15 | +67% | Few weeks after Fusaka |\\n| **BPO 2** | 14 | 21 | +40% | Few weeks after BPO 1 |\\n\\n## Thanks\\n\\nShoutout to the client teams who worked closely with us throughout this devnet, and to the [Sunnyside Labs](https://x.com/testinprod_io) team who ran a large amount of nodes!\\n\\n<div style={{ textAlign: \'center\', marginTop: \'40px\', fontSize: \'1.5rem\', fontWeight: \'bold\' }}>\\n \ud83d\ude80\ud83d\ude80\ud83d\ude80\\n</div>"},{"id":"gaslimit-scaling","metadata":{"permalink":"/posts/gaslimit-scaling","source":"@site/blog/2025-06-30-gaslimit-scaling/index.md","title":"Scaling Ethereum: The Path to 45M Gas Limit and Beyond","description":"An analysis of Ethereum\u2019s gas limit scaling strategy through performance testing and security analysis","date":"2025-06-30T00:00:00.000Z","tags":[{"inline":true,"label":"ethereum","permalink":"/posts/tags/ethereum"},{"inline":true,"label":"gaslimit","permalink":"/posts/tags/gaslimit"},{"inline":true,"label":"scaling","permalink":"/posts/tags/scaling"}],"readingTime":6.34,"hasTruncateMarker":false,"authors":[{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null},{"name":"pk910","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/pk910","github":"https://github.com/pk910","twitter":"https://x.com/_pk910_","imageURL":"/img/team/pk910.jpeg","key":"pk910","page":null}],"frontMatter":{"slug":"gaslimit-scaling","title":"Scaling Ethereum: The Path to 45M Gas Limit and Beyond","authors":["parithosh","pk910"],"description":"An analysis of Ethereum\u2019s gas limit scaling strategy through performance testing and security analysis","tags":["ethereum","gaslimit","scaling"],"image":"/img/blog/gaslimit-panda.png"},"unlisted":false,"prevItem":{"title":"Fusaka-Devnet-5 BPO Analysis","permalink":"/posts/fusaka-devnet-5-bpo-analysis"},"nextItem":{"title":"EIP-7691 Retrospective","permalink":"/posts/eip7691-retrospective"}},"content":"## Introduction\\n\\nAs Ethereum continues to grow, the demand for block space has grown with it. While L2s continue to scale, we are committing to scale the L1 as well. With the successful deployment of Pectra and the increase from 3/6 to 6/9 blobs, the network has demonstrated its ability to scale blobs systematically while maintaining security and decentralization. The next fork, Fusaka, ships a feature (PeerDAS) that is aimed at improving blob throughput. With progress being made on that front, we turn our attention to the other frontier: scaling the gas limit from today\'s 36M to 45M in the near term, with a clear path toward 60M and later 100M gas limit per block.\\n\\nThis post outlines our comprehensive approach to gas limit scaling, informed by extensive testing conducted during the Berlin Interop week and ongoing research efforts in bottlenecks.\\n\\n## Background: Why Gas Limit Scaling is hard\\n\\nThe gas limit directly determines Ethereum\'s transaction throughput capacity and has a lot of second order effects on the network. However, scaling the gas limit isn\'t simply a matter of changing a parameter\u2014it requires careful analysis of:\\n\\n- **Execution performance**: How quickly can clients process larger blocks?\\n- **Network propagation**: Can blocks reach all the nodes in the network (validators and full nodes) within the 4-second deadline?\\n- **State growth**: How do larger blocks affect sync times, data access times and storage requirements?\\n- **RPC Performance:** Can we continue to serve the data at the tip of the chain?\\n- **Consensus layer impacts**: What are the implications for consensus operations?\\n\\n## Our Three Pillar Approach\\n\\n### 1. OPCODE and Pre
1compile Benchmarking\\n\\nExecuting OPCODES and Precompiles directly influence the amount of time it takes to execute a block. If transactions takes longer to validate than the attestation deadline, then we run into higher chances of the block being missed by a portion of the network - allowing for a straightforward attack. Such transactions imply that either we have priced the gas required to execute them wrong, or that client teams are missing optimisations to improve performance.\\n\\n**Current Status**: Significant progress has been achieved during the Berlin Interop week. Client teams have improved their worstcase block processing times substantially, with most operations now meeting our 20M gas/second threshold (gas/second is expanded on in the footnote).\\n\\n**Key Findings**:\\n\\n- ECRecover, Blake2f, Datacopy, BLS precompiles, and MSTORE are all non-issues at scale\\n- BN256Add, BN256Pairing, and BN256Mul were successfully optimized by client teams\\n- The primary remaining bottleneck is MODEXP, which will be addressed by EIP-7883 and by further client improvements\\n\\nDashboard showing worst case gas performance (after MODEXP repricing):\\n\\n\\n\\n\\n**Testing Infrastructure**: Our performance testing leverages tooling including:\\n- **[perf-devnet-1](https://ethpandaops.io/networks/perf-devnet-1/)**: A mainnet shadowfork running at 100M gas with realistic loads, allowing us to simulate mainnet-like performance on transaction loads\\n- **[gas-benchmarks](https://github.com/NethermindEth/gas-benchmarks)**: Automated benchmark framework, allowing us to reproduce issues and compare cached and uncached operations\\n- **[execution-spec-tests](https://github.com/ethereum/execution-spec-tests)**: The canonical method of writing the performance tests that can be consumed by the benchmarking tool as well as client CIs.\\n\\n### 2. State Growth Analysis\\n\\nThe higher the gas limit, the higher the potential for rate of state growth to increase. A larger state would imply it takes longer to sync a node, however, it also takes a toll on database performance once the state is significantly larger. More concerns about state growth are covered in this video \\"[Ethereum in numbers: Where TPS meets physics / P\xe9ter Szil\xe1gyi](https://www.youtube.com/watch?v=Cmuz_Xn_YJw)\\". Hence, our scaling approaches need to pay extreme attention to state growth and its longer term implications.\\n\\n**Approach**: We\'re conducting comprehensive analysis of how larger blocks affect state growth, processing degradations and sync performance using specific devnets with significantly larger state sizes.\\n\\n**Key Concerns**:\\n- How do deeper trie structures affect Storage operations (like SSTORE,Contract deployments, SLOAD, witness ) at scale?\\n- How does larger state affect trie root calculation?\\n- How do the various database styles client teams use perform at these scales?\\n- What are the implications for snap sync performance?\\n- How do archive nodes handle increased storage requirements?\\n- Can we continue to serve RPC requests in a reasonable time with higher gas limits and larger state?\\n\\n**Testing Environment**:\\n- **Bloatnet**: Specialized devnet generating 2x mainnet state size with deep leaf inclusions\\n- Real world transaction patterns to simulate authentic growth scenarios\\n- Comprehensive RPC performance testing under various load conditions\\n\\n### 3. Security and Consensus Layer Implications\\n\\nGas limit increases don\'t occur in isolation - they have profound implications for the consensus layer and overall network security. The consensus layer currently has a [10MB p2p gossip limit](https://github.com/ethereum/consensus-specs/blob/815751ecb3debc11c47aa67ffbf2ecde1c941c4e/specs/phase0/p2p-interface.md?plain=1#L217), so we need to ensure that the overall block size along with the worst case consensus layer overheads (Attester slashings) never exceed this threshold. Larger block sizes also imply a worsening in propagation times, we would need to ensure that the p99 of the network continues to receive the blocks in a timely manner.\\n\\n**Approach**: We\'re spending a lot of time analysing gas limits from a security perspective and analysing network performance in order to ensure the network can handle higher gas limits.\\n\\n**Key Considerations**:\\n- **[Beacon block size implications](https://ethpandaops.github.io/blocksize/)**: We need to ensure that the combined worst case sizes of all the layers don\'t exceed some p2p thresholds.\\n- **[Network propagation](https://ethpandaops.io/posts/eip7691-retrospective/)**: Analysis ensuring blocks reach supermajority within 4-second deadlines\\n\\n## Current recommended milestone: 45M\\n\\n### 45M Gas Limit (Immediate)\\n\\n**Status**: **Ready for deployment!**\\n\\
1nOur comprehensive testing shows no blockers for moving to 45M gas limit. All client implementations demonstrate adequate performance, and network propagation remains well within safety margins.\\n\\n**Key Metrics**:\\n- Worstcase block processing times: <3 seconds across all major clients\\n- Network propagation: 99.5% of nodes achieve <4s acceptance rates\\n- No additional EIPs required\\n- All clients have released updated defaults (or will do so soon)\\n\\n## Path forward\\n\\nThe path to scaling doesn\'t stop at 45M, we intend to use the tools to continue scaling Ethereum safely. We have already identified some MODEXP related optimisations that could get us some more short term scaling, with Elliptic Curve related optimisations required for scaling beyond that. We also will have the Fusaka fork later this year that will theoretically enable scaling up to a gas limit of 100M.\\n\\nWe are hoping that the groundwork we lay for gas limit increases help not just the L1 scaling journey, but also the L2 scaling journey. The aim is for the tools, approaches and analyses we release over the next months to help the entire Ethereum community.\\n\\nWe do want to reiterate, as we continue this journey, our commitment remains unchanged: scale Ethereum responsibly, maintain decentralization, and never compromise security for throughput. The Berlin Interop week demonstrated the Ethereum ecosystem\'s remarkable ability to collaborate on performance improvements - so we\'d like to give a massive shoutout to all the client teams that rallied to make this happen.\\n\\n#### Footnote\\n\\nA slot takes 12s on Ethereum, however, we have an attestation deadline at 4s. This implies that the block needs to be produced and propagated within 4s mark under ideal operation. We talk about gas/second while talking about benchmarks as they are a way to represent how large of a block we can accept within the 4s attestation deadline of a slot. If we have a client perform at 20M gas/second, that implies that the maximum the network can handle is a gas limit of 80M per block -> 20M gas/sec x 4s."},{"id":"eip7691-retrospective","metadata":{"permalink":"/posts/eip7691-retrospective","source":"@site/blog/2025-05-30-7691-retrospective/index.md","title":"EIP-7691 Retrospective","description":"Deep dive into EIP-7691 and the impact on the network","date":"2025-05-30T00:00:00.000Z","tags":[{"inline":true,"label":"ethereum","permalink":"/posts/tags/ethereum"},{"inline":true,"label":"eip7691","permalink":"/posts/tags/eip-7691"},{"inline":true,"label":"analysis","permalink":"/posts/tags/analysis"}],"readingTime":12.21,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"eip7691-retrospective","title":"EIP-7691 Retrospective","authors":["samcm"],"description":"Deep dive into EIP-7691 and the impact on the network","tags":["ethereum","eip7691","analysis"],"image":"/img/blog/eip7691-retrospective/hero.png","githubRepos":[{"name":"xatu-analysis","url":"https://github.com/ethpandaops/xatu-analysis"}],"relatedLinks":[{"name":"EthResearch post","url":"https://ethresear.ch/t/block-arrivals-home-stakers-bumping-the-blob-count/21096"},{"name":"Worst case block sizes","url":"https://hackmd.io/@Nerolation/H1bCu_Z2kl"}]},"unlisted":false,"prevItem":{"title":"Scaling Ethereum: The Path to 45M Gas Limit and Beyond","permalink":"/posts/gaslimit-scaling"},"nextItem":{"title":"60M Gas Limit on Sepolia & Hoodi","permalink":"/posts/60m-gas-sepolia-hoodi"}},"content":"## Introduction\\n\\n[EIP-7691](https://eips.ethereum.org/EIPS/eip-7691) was recently enabled on Mainnet with the Pectra hardfork, bumping the blob count from 3 Target 6 Max, to 6 Target 9 Max. This analysis focuses on the results of this change on the network, asserting that our previous analysis that justified the increase was correct.\\n\\n## Background\\n\\nWhen a block is proposed, the proposer needs to quickly gossip the block and all accompanying blobs to at least 66% of the network within 4 seconds, otherwise they risk their block becoming orphaned. The [previous anal
1ysis](https://ethresear.ch/t/block-arrivals-home-stakers-bumping-the-blob-count/21096) primarily looked at this scenario from two angles:\\n - Solo stakers proposing locally built blocks.\\n - Home users observing these blocks.\\n\\nThis approach assumed these two groups represented the nodes most sensitive to an increase in bandwidth requirements.\\n\\n## Methodology\\n\\n### Data Sources\\n\\nOur analysis leverages data from the ethPandaOps monitoring infrastructure, which provides visibility into Ethereum network performance through the [Xatu](/data/xatu) data collection system. Nodes are run by both the ethPandaOps team and the community. Xatu connects to the Beacon API event stream, which has no guarantees on event publication or ordering, and as such can be considered _worst_ case performance.\\n\\n### Parameters\\n\\nWe analyzed Mainnet over a 7-day period to understand the impact:\\n\\nexport const testnetData = [\\n {\\n name: \\"Mainnet\\",\\n emoji: \\"\ud83d\udc51\\",\\n description: \\"The Layer 1 we all know and love\\",\\n stats: [\\n { icon: \\"\ud83d\udcc5\\", label: \\"Period\\", value: \\"May 21-28, 2025\\" },\\n { icon: \\"\ud83d\udd32\\", label: \\"Slots\\", value: \\"50,025\\" },\\n { icon: \\"\ud83d\udce1\\", label: \\"Nodes\\", value: \\"123 (29 ethPandaOps, 94 home user)\\" },\\n { icon: \\"\ud83c\udf0d\\", label: \\"Countries\\", value: \\"27\\" },\\n { icon: \\"\ud83d\udd27\\", label: \\"Consensus Layer Clients\\", value: \\"6\\" }\\n ]\\n },\\n];\\n\\n<div style={{ display: \'grid\', gridTemplateColumns: \'repeat(auto-fit, minmax(300px, 1fr))\', gap: \'20px\', marginTop: \'20px\' }}>\\n {testnetData.map((testnet, index) => (\\n <div key={index} style={{\\n backgroundColor: \'var(--ifm-color-emphasis-100)\',\\n borderRadius: \'8px\',\\n padding: \'20px\',\\n border: \'1px solid var(--ifm-color-emphasis-200)\'\\n }}>\\n <h4 style={{ marginTop: 0, marginBottom: \'15px\', color: \'var(--ifm-color-primary)\' }}>\\n {testnet.emoji} {testnet.name}\\n </h4>\\n <p style={{ fontSize: \'0.9rem\', marginBottom: \'15px\', color: \'var(--ifm-color-content-secondary)\' }}>\\n {testnet.description}\\n </p>\\n <div style={{ display: \'grid\', gap: \'10px\' }}>\\n {testnet.stats.map((stat, statIndex) => (\\n <div key={statIndex} style={{\\n display: \'flex\',\\n justifyContent: \'space-between\',\\n borderBottom: statIndex < testnet.stats.length - 1 ? \'1px solid var(--ifm-color-emphasis-200)\' : \'none\',\\n paddingBottom: \'5px\'\\n }}>\\n <span style={{ fontWeight: 600 }}>{stat.icon} {stat.label}</span>\\n <span>{stat.value}</span>\\n </div>\\n ))}\\n </div>\\n </div>\\n ))}\\n</div>\\n\\n### Metrics\\n\\nOur analysis focuses on our \\"New Head\\" metric. For those that read our previous blog post on the [60M Gas Limit on Sepolia & Hoodi](/posts/60m-gas-sepolia-hoodi), this is the same metric that we used in that post as well.\\n\\n\\n\\n<div style={{ \\n background: \'var(--ifm-color-emphasis-100)\',\\n borderRadius: \'12px\',\\n padding: \'40px 20px\',\\n marginBottom: \'30px\'\\n}}>\\n <h4 style={{ marginTop: 0, marginBottom: \'40px\', textAlign: \'center\', fontSize: \'1.2rem\' }}>\u23f1\ufe0f Block Production Timeline</h4>\\n \\n {/* Add custom styles for mobile responsiveness */}\\n <style>{`\\n @media (max-width: 768px) {\\n .timeline-desktop { display: none !important; }\\n .timeline-mobile { display: block !important; }\\n }\\n @media (min-width: 769px) {\\n .timeline-desktop { display: block !important; }\\n .timeline-mobile { display: none !important; }\\n }\\n .timeline-arrow {\\n animation: slideRight 2s ease-in-out infinite;\\n }\\n @keyframes slideRight {\\n 0%, 100% { transform: translateX(0); }\\n 50% { transform: translateX(10px); }\\n }\\n `}</style>\\n \\n {/* Desktop Timeline */}
1\\n <div className=\\"timeline-desktop\\" style={{ maxWidth: \'800px\', margin: \'0 auto\' }}>\\n \\n {/* Timeline bar */}\\n <div style={{ position: \'relative\', marginBottom: \'60px\' }}>\\n {/* Background bar */}\\n <div style={{\\n position: \'absolute\',\\n left: \'0\',\\n right: \'0\',\\n top: \'20px\',\\n height: \'4px\',\\n background: \'var(--ifm-color-emphasis-300)\',\\n borderRadius: \'2px\'\\n }} />\\n \\n {/* Arrow indicators */}\\n <div style={{\\n position: \'absolute\',\\n right: \'-5px\',\\n top: \'16px\',\\n width: \'0\',\\n height: \'0\',\\n borderLeft: \'12px solid var(--ifm-color-emphasis-300)\',\\n borderTop: \'6px solid transparent\',\\n borderBottom: \'6px solid transparent\'\\n }} />\\n \\n {/* Time flow indicator */}\\n <div style={{\\n position: \'absolute\',\\n left: \'50%\',\\n top: \'-15px\',\\n transform: \'translateX(-50%)\',\\n fontSize: \'0.75rem\',\\n color: \'var(--ifm-color-emphasis-600)\',\\n fontWeight: \'bold\',\\n display: \'flex\',\\n alignItems: \'center\',\\n gap: \'5px\'\\n }}>\\n Time \u2192\\n </div>\\n \\n {/* Timeline points */}\\n <div style={{ position: \'relative\', display: \'flex\', justifyContent: \'space-between\' }}>\\n {/* T=0 */}\\n <div style={{ flex: \'0 0 80px\', textAlign: \'center\' }}>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'var(--ifm-color-primary)\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n color: \'white\',\\n fontWeight: \'bold\',\\n fontSize: \'0.9rem\'\\n }}>\ud83d\udce4</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\' }}>Block<br/>Published</div>\\n </div>\\n \\n {/* Processing */}\\n <div style={{ flex: \'0 0 100px\', textAlign: \'center\' }}>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'#ffc107\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n fontSize: \'1.2rem\'\\n }}>\u2699\ufe0f</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\' }}>Execute &<br/>Propagate</div>\\n </div>\\n \\n {/* Metric 1 */}\\n <div style={{ flex: \'0 0 140px\', textAlign: \'center\', position: \'relative\' }}>\\n {/* Highlight box */}\\n <div style={{\\n position: \'absolute\',\\n top: \'-10px\',\\n left: \'50%\',\\n transform: \'translateX(-50%)\',\\n background: \'#28a745\',\\n color: \'white\',\\n padding: \'4px 12px\',\\n borderRadius: \'20px\',\\n fontSize: \'0.7rem\',\\n fontWeight: \'bold\',\\n whiteSpace: \'nowrap\',\\n boxShadow: \'0 2px 4px rgba(0,0,0,0.2)\'\\n }}>\ud83d\udcca KEY METRIC</div>\\n \\n <div style={{\\n width: \'50px\',\\n height: \'50px\',\\n borderRadius: \'50%\',\\n background: \'#28a745\',\\n margin: \'20px auto 0\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n color: \'white\',\\n fontSize: \'1.5rem\',\\n boxShadow: \'0 4px 8px rgba(40, 167, 69, 0.3)\',\\n border: \'3px solid rgba(40, 167, 69, 0.2)\'\\n }}>\u2713</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.85rem\', fontWeight: \'bold\', color: \'#28a745\' }}>New Head<br/>Metric</div>\\n </div>\\n \\n {/* Deadline */}\\n <div style={{ flex: \'0 0 80px\', textAlign: \'center\' }}>\\n <div style={{ fontSize: \'0.7rem\', color: \'#dc3545\', marginBottom: \'5px\', fontWeight: \'bold\' }}>4 seconds</div>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'#dc3545\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n color: \'white\',\\n fontWeight: \'bold\',\\n fontSize: \'0.9rem\'\\n }}>\u23f0</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\', color: \'#dc3545\' }}>Attestation<br/>Deadline</div>\\n </div>\\n </div>\\n </div>\\n </div>\\n \\n {/* Mobile Timeline - Simple vertical list */}
1\\n <div className=\\"timeline-mobile\\" style={{ display: \'none\', padding: \'0 20px\' }}>\\n <div style={{ \\n display: \'flex\', \\n alignItems: \'center\', \\n justifyContent: \'center\',\\n marginBottom: \'20px\',\\n fontSize: \'0.85rem\',\\n color: \'var(--ifm-color-emphasis-600)\',\\n fontWeight: \'bold\'\\n }}>\\n \u2193 Time flows downward \u2193\\n </div>\\n <div style={{ display: \'flex\', flexDirection: \'column\', gap: \'20px\' }}>\\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid var(--ifm-color-primary)\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\' }}>\ud83d\udce4 Block Published</div>\\n <div>Block is published to the network</div>\\n </div>\\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid #ffc107\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\' }}>\u2699\ufe0f Processing</div>\\n <div>Execute block & propagate blobs</div>\\n </div>\\n \\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid #28a745\',\\n position: \'relative\',\\n boxShadow: \'0 2px 8px rgba(40, 167, 69, 0.15)\'\\n }}>\\n <div style={{\\n position: \'absolute\',\\n top: \'-10px\',\\n left: \'15px\',\\n background: \'#28a745\',\\n color: \'white\',\\n padding: \'2px 10px\',\\n borderRadius: \'12px\',\\n fontSize: \'0.65rem\',\\n fontWeight: \'bold\'\\n }}>\ud83d\udcca KEY METRIC</div>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\', color: \'#28a745\', marginTop: \'10px\' }}>\u2713 New Head Metric</div>\\n <div>Block becomes new chain head</div>\\n </div>\\n \\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid #dc3545\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\', color: \'#dc3545\' }}>\u23f0 4s Deadline</div>\\n <div>Attestation deadline - must reach 66% of network</div>\\n </div>\\n </div>\\n </div>\\n \\n {/* Metric descriptions */}\\n <div style={{ display: \'grid\', gridTemplateColumns: \'repeat(auto-fit, minmax(280px, 1fr))\', gap: \'20px\', marginTop: \'50px\' }}>\\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'20px\', \\n borderRadius: \'8px\',\\n border: \'2px solid #28a745\',\\n boxShadow: \'0 2px 4px rgba(0,0,0,0.05)\'\\n }}>\\n <h5 style={{ marginTop: 0, color: \'#28a745\', fontSize: \'1rem\' }}>\u2713 New Head</h5>\\n <p style={{ fontSize: \'0.85rem\', marginBottom: \'10px\' }}>When the client considers the block as the new chain head</p>\\n <ul style={{ marginBottom: 0, fontSize: \'0.8rem\', paddingLeft: \'20px\' }}>\\n <li>Formula: <code style={{ fontSize: \'0.75rem\' }}>max(block_event, head_event, blobs_events...)</code></li>\\n <li>Includes execution time</li>\\n <li>Must wait for all blobs</li>\\n </ul>\\n </div>\\n </div>\\n</div>\\n\\n## Analysis\\n\\nSince the [ethresear.ch post](https://ethresear.ch/t/block-arrivals-home-stakers-bumping-the-blob-count/21096) that justified the increase, the Xatu public contributor system has expanded from ~20 nodes to ~80 nodes. We\'ve gone from 9 countries to 27, and now cover 52 distinct cities. There has also been a recent push to pump the gas limit, with Mainnet currently running at 36M compared to 30M at the time of the previous analysis.\\n\\nGiven these changes, our analysis foundation has slightly changed.\\n\\n1. The geographical distribution of nodes contributing to Xatu has shifted, with more nodes in more countries. This is especially important as one of the bottlenecks to block propagation is the speed of light and the time it takes for data to travel across the globe.\\n1. The worst-case block size has slightly increased, as the gas limit has increased.\\n1. All nodes will take slightly longer to process the block, as the gas limit has increased.\\n\\n---\\n\\n### 1. Was 6/9 safe for home users?\\n We can observe our <span style={{ color: \'#28a745\', fontWeight: \'bold\' }}>\u2713 New Head</span> metric for all Locally Built blocks proposed by Solo Stakers and observed by Home Users.\\n\\n<ClickableImage\\n src=\\"/img/blog/eip7691-retrospective/heatmap.png\\"\\n alt=\\"Mainnet Figure 1\\"\\n caption=\\"Figure 1: Mainnet (\u2713 New Head metric) against combined block+blobs size (compressed). Each point represents a \'New Head\' observation by a Xatu node.\\"\\n/>
1\\nWe observe that our home users have been able to support the increase blob count, with only a few outliers outside of the 4s threshold. We can also run the same analysis that locked in 6/9 to see what the predicted maximum blob count is today:\\n\\n<ClickableImage\\n src=\\"/img/blog/eip7691-retrospective/figure-1.png\\"\\n alt=\\"Mainnet Figure 2\\"\\n caption=\\"Figure 2: Predicted Max Blob Count for Mainnet (\u2713 New Head metric) with historical p99 block size. Each point represents a \'New Head\' observation by a Xatu node.\\"\\n/>\\n\\nThose with a keen eye will notice that this configuration seemingly supports up to 20 blobs! Unfortunately this is not the full picture as we have to account for the worst case scenario block size on top of our 9 blobs.\\n\\n<ClickableImage\\n src=\\"/img/blog/eip7691-retrospective/figure-2.png\\"\\n alt=\\"Mainnet Figure 3\\"\\n caption=\\"Figure 3: Predicted Maximum Blob Count for Mainnet (\u2713 New Head metric) with worst case block size at a 36M gas limit. Each point represents a \'New Head\' observation by a Xatu node.\\"\\n/>\\n\\nThis configuration more accurately reflects the restrictions of today, and we can observe a predicted maximum blob count of 14, meaning that 6/9 was safe for home users.\\n\\n\\n<Alert type=\\"success\\">\\n Outcome: <strong>Home users were able to support 9 blobs \ud83c\udf89</strong>\\n</Alert>\\n\\n\\n### 2. Can we support a 60M gas limit?\\n\\nLet\'s first take a look at the worst case scenario block size at a 60M gas limit in the same configuration.\\n\\n<ClickableImage\\n src=\\"/img/blog/eip7691-retrospective/figure-3.png\\"\\n alt=\\"Mainnet Figure 4\\"\\n caption=\\"Figure 4: Predicted Maximum Blob Count for Mainnet (\u2713 New Head metric) with worst case block size at a 60M gas limit. Each point represents a \'New Head\' observation by a Xatu node.\\"\\n/>\\n\\nCutting it a tad close now! A worst-case block size with 60M gas used only supports 10 blobs, so the 6/9 configuration has really hit the mark for our home users. As a reminder, these are all worst case scenarios, but it\'s reassuring to see that we can compound these types of scenarios and still be able to support our most bandwidth sensitive nodes.\\n\\n*Note: this highlights the importance of limiting the gas limit in Pectra to 60M.*\\n\\n<Alert type=\\"success\\">\\n Outcome: <strong>60M gas limit is safe for home user building local blocks at 6/9 \ud83c\udf89</strong>\\n</Alert>\\n\\n### 3. What about when using MEV Boost?\\n\\nLet\'s take a look at historical p99 block size but for blocks that are provided by MEV Relays. This is a good baseline as [~91% of blocks are provided by MEV Relays](https://mevboost.pics/).\\n\\n<ClickableImage\\n src=\\"/img/blog/eip7691-retrospective/home-users-mev-p99-block-size.png\\"\\n alt=\\"Mainnet Figure 5\\"\\n caption=\\"Figure 5: Predicted Maximum Blob Count for Mainnet (\u2713 New Head metric) with p99 historical block size, MEV Relay blocks. Each point represents a \'New Head\' observation by a Xatu node.\\"\\n/>\\n\\nAround the same as locally built blocks, which is reassuring. Now lets look at the worst case scenario block size at a 60M gas limit.\\n\\n<ClickableImage\\n src=\\"/img/blog/eip7691-retrospective/home-users-mev-60m.png\\"\\n alt=\\"Mainnet Figure 6\\"\\n caption=\\"Figure 6: Predicted Maximum Blob Count for Mainnet (\u2713 New Head metric) with worst case block size at a 60M gas limit, MEV Relay blocks. Each point represents a \'New Head\' observation by a Xatu node.\\"\\n/>\\n\\nAt first glance this is a worrying result, with the predicted maximum supported blob count being 5! However this is a worst case scenario block size under 60M gas. Some relays have started delaying header responses (playing timing games), which is highly likely to be the cause of this. We would expect relays to adjust their timing games as the landscape evolves, rendering this forward-looking analysis of the network a little moot.\\n\\nTo investigate our MEV Relay findings, we can view the data differently, opting for the distribution of when slots are \'accepted\' by the Xatu network. That\'s a fancy way of saying:\\n- For every slot, find the time where 66% of the Xatu nodes saw the block as the new chain head.\\n- Plot the CDF of this metric.\\n\\n<ClickableImage\\n src=\\"/img/blog/eip7691-retrospective/cdf.png\\"\\n alt=\\"Mainnet Figure 7\\"\\n caption=\\"Figure 7: Mainnet CDF of p66(\u2713 New Head) per slot in various configurations.\\"\\n/>\\n\\nWe observe that locally built blocks (dashed lines) consistently achieve faster acceptance times compared to MEV Relay blocks (solid lines). This aligns with expectations, given locally built blocks don\u2019t suffer the overhead of the MEV bid-acceptance dance. Amongst the MEV data is also _operators_ playing their own [timing games](https://timings.pics).\\n\\nHigh-bandwidth ethPandaOps nodes reach \\\\<4s acceptance rates of 99.5%, compared to just 97.1% for home user nodes. This discrepancy likely boils down to bandwidth limitations, and is potentially compounded by a few outliers in the home user group (...like our team mate, Barnabas, whose setup might run on a 1990s spinning HDD and vibes). These potential outliers will need further investigation as we push on with PeerDAS in Fusaka.\\n\\n<Alert type=\\"warning\\">\\n Outcome: <strong>As expected; needs monitoring</strong>\\n</Alert>\\n\\n## Conclusion\\n\\nOur evaluation confirms that 6/9 was right on the money for home users, and our data demonstrates that a 60M gas limit remains viable even under worst-case block size conditions. However, we found that a 60M gas limit appears to be about the limit of what home users can support on the Pectra fork.\\n\\nInsights gathered from MEV Boost blocks surfaced the inherent issues of forward-looking analysis, but generally aligned with our expectations. The team is now heads-down on Fusaka, which includes PeerDAS, significantly enhancing blob scalability for our Layer 2 friends.\\n\\n<div style={{ textAlign: \'center\', marginTop: \'40px\', fontSize: \'1.5rem\', fontWeight: \'bold\' }}>\\n \ud83d\ude80\ud83d\ude80\ud83d\ude80\\n</div>\\n\\n## Acknowledgments\\n\\nSpecial thanks to our contributoor community for providing the distributed monitoring data that makes this analysis possible. Their participation gives us unprecedented visibility into Ethereum network behavior.\\n\\n_Want to contribute to our monitoring efforts? [Become a contributoor!](/contribute-data/)_"},{"id":"60m-gas-sepolia-hoodi","metadata":{"permalink":"/posts/60m-gas-sepolia-hoodi","source":"@site/blog/2025-05-27-60m-gas-sepolia-hoodi/index.md","title":"60M Gas Limit on Sepolia & Hoodi","description":"Deep dive into network performance metrics after bumping the gas limit to 60M on Sepolia and Hoodi testnets","date":"2025-05-27T00:00:00.000Z","tags":[{"inline":true,"label":"ethereum","permalink":"/posts/tags/ethereum"},{"inline":true,"label":"sepolia","permalink":"/posts/tags/sepolia"},{"inline":true,"label":"hoodi","permalink":"/posts/tags/hoodi"},{"inline":true,"label":"analysis","permalink":"/posts/tags/analysis"}],"readingTime":10.47,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","t
1witter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"60m-gas-sepolia-hoodi","title":"60M Gas Limit on Sepolia & Hoodi","authors":["samcm"],"description":"Deep dive into network performance metrics after bumping the gas limit to 60M on Sepolia and Hoodi testnets","tags":["ethereum","sepolia","hoodi","analysis"],"image":"/img/blog/60m-gas-testnets.png","githubRepos":[{"name":"xatu-analysis","url":"https://github.com/ethpandaops/xatu-analysis"}],"relatedLinks":[{"name":"Gas Limit Tracker","url":"https://gaslimit.pics"}]},"unlisted":false,"prevItem":{"title":"EIP-7691 Retrospective","permalink":"/posts/eip7691-retrospective"},"nextItem":{"title":"Pectra Mainnet Checklist","permalink":"/posts/pectra-mainnet-checklist"}},"content":"## Introduction\\n\\nWith Pectra out the door, and [EIP7623](https://eips.ethereum.org/EIPS/eip-7623) limiting the worst case execution block size, we now have more room to safely increase the gas limit on Mainnet. [153,000](https://gaslimit.pics) validators now signal for a 60M gas limit on Mainnet, and core developers recently bumped the gas limit to 60M on the Sepolia and Hoodi testnets to analyze the impact of this change.\\n\\nThis analysis will focus on the impact of this change on network propagation, block processing, and overall performance on these testnets. While it\'s unlikely that we will see any significant impact on these testnets due to the different network topology and validator set, this analysis is useful for ensuring that clients are able to handle the increased gas limit on a fundamental level.\\n\\n## Background\\n\\nThe gas limit determines the maximum amount of computational work that can be included in a single block. However, this comes at a cost to _all_ participants in the network.\\n\\n1. Validators have higher hardware requirements as they need to build and publish a larger block in the same amount of time.\\n2. All nodes on the network must then download and execute the larger block, increasing network and system load.\\n\\n## Methodology\\n\\n### Data Sources\\n\\nOur analysis leverages data from the ethPandaOps monitoring infrastructure, which provides visibility into Ethereum network performance through the [Xatu](/data/xatu) data collection system. Nodes are run by both the ethPandaOps team and the community. Xatu connects to the Beacon API event stream, which has no guarantees on event publication or ordering, and as such can be considered _worst_ case performance.\\n\\n\\n### Parameters\\n\\nWe analyzed Sepolia & Hoodi over a 14-day period to understand the impact of the 60M gas limit increase:\\n\\nexport const testnetData = [\\n {\\n name: \\"Sepolia Testnet\\",\\n emoji: \\"\ud83d\udd35\\",\\n description: \\"Permissioned testnet with a closed validator set and large execution state size, making it more representative of mainnet conditions.\\",\\n stats: [\\n { icon: \\"\ud83d\udcc5\\", label: \\"Period\\", value: \\"May 13-27, 2025\\" },\\n { icon: \\"\ud83d\udd32\\", label: \\"Blocks\\", value: \\"91,957\\" },\\n { icon: \\"\ud83d\udce1\\", label: \\"Nodes\\", value: \\"16 Xatu nodes\\" },\\n { icon: \\"\ud83c\udf0d\\", label: \\"Regions\\", value: \\"EU, NA, Oceania\\" },\\n { icon: \\"\ud83d\udd27\\", label: \\"Clients\\", value: \\"5 CL implementations\\" }\\n ]\\n },\\n {\\n name: \\"Hoodi Testnet\\",\\n emoji: \\"\ud83d\udfe3\\",\\n description: \\"Newer testnet with an open validator set and smaller execution state size, providing a different performance profile.\\",\\n stats: [\\n { icon: \\"\ud83d\udcc5\\", label: \\"Period\\", value: \\"May 13-27, 2025\\" },\\n { icon: \\"\ud83d\udd32\\", label: \\"Blocks\\", value: \\"89,065\\" },\\n { icon: \\"\ud83d\udce1\\", label: \\"Nodes\\", value: \\"21 Xatu nodes\\" },\\n { icon: \\"\ud83c\udf0d\\", label: \\"Regions\\", value: \\"EU, NA, Oceania\\" },\\n { icon: \\"\ud83d\udd27\\", label: \\"Clients\\", value: \\"5 CL implementations\\" }\\n ]\\n }\\n];\\n\\n<div style={{ display: \'grid\', gridTemplateColumns: \'repeat(auto-fit, minmax(300px, 1fr))\', gap: \'20px\', marginTop: \'20px\' }}>\\n {testnetData.map((testnet, index) => (\\n <div key={index} style={{\\n backgroundColor: \'var(--ifm-color-emphasis-100)\',\\n borderRadius: \'8px\',\\n padding: \'20px\',\\n border: \'1px solid var(--ifm-color-emphasis-200)\'\\n }}>\\n <h4 style={{ marginTop: 0, marginBottom: \'15px\', color: \'var(--ifm-color-primary)\' }}>\\n {testnet.emoji} {testnet.name}\\n </h4>\\n <p style={{ fontSize: \'0.9rem\', marginBottom: \'15px\', color: \'var(--ifm-color-content-secondary)\' }}>\\n {testnet.description}\\n </p>\\n <div style={{ display: \'grid\', gap: \'10px\' }}>\\n {testnet.stats.map((stat, statIndex) => (\\n <div key={statIndex} style={{\\n display: \'flex\',\\n justifyContent: \'space-between\',\\n borderBottom: statIndex < testnet.stats.length - 1 ? \'1px solid var(--ifm-color-emphasis-200)\' : \'none\',\\n paddingBottom: \'5px\'\\n }}>
1\\n <span style={{ fontWeight: 600 }}>{stat.icon} {stat.label}</span>\\n <span>{stat.value}</span>\\n </div>\\n ))}\\n </div>\\n </div>\\n ))}\\n</div>\\n\\n### Metrics\\n\\nOur analysis will focus on 2 metrics that fire at different points in the slot lifecycle, comparing them against Gas Usage of the block.\\n\\n<div style={{ \\n background: \'var(--ifm-color-emphasis-100)\',\\n borderRadius: \'12px\',\\n padding: \'40px 20px\',\\n marginBottom: \'30px\'\\n}}>\\n <h4 style={{ marginTop: 0, marginBottom: \'40px\', textAlign: \'center\', fontSize: \'1.2rem\' }}>\u23f1\ufe0f Block Production Timeline</h4>\\n \\n {/* Add custom styles for mobile responsiveness */}\\n <style>{`\\n @media (max-width: 768px) {\\n .timeline-desktop { display: none !important; }\\n .timeline-mobile { display: block !important; }\\n }\\n @media (min-width: 769px) {\\n .timeline-desktop { display: block !important; }\\n .timeline-mobile { display: none !important; }\\n }\\n `}</style>\\n \\n {/* Desktop Timeline */}
1\\n <div className=\\"timeline-desktop\\" style={{ maxWidth: \'800px\', margin: \'0 auto\' }}>\\n \\n {/* Timeline bar */}\\n <div style={{ position: \'relative\', marginBottom: \'60px\' }}>\\n {/* Background bar */}\\n <div style={{\\n position: \'absolute\',\\n left: \'0\',\\n right: \'0\',\\n top: \'20px\',\\n height: \'4px\',\\n background: \'var(--ifm-color-emphasis-300)\',\\n borderRadius: \'2px\'\\n }} />\\n \\n {/* Timeline points */}\\n <div style={{ position: \'relative\', display: \'flex\', justifyContent: \'space-between\' }}>\\n {/* T=0 */}\\n <div style={{ flex: \'0 0 80px\', textAlign: \'center\' }}>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'var(--ifm-color-primary)\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n color: \'white\',\\n fontWeight: \'bold\',\\n fontSize: \'0.9rem\'\\n }}>?</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\' }}>Block<br/>Published</div>\\n </div>\\n \\n {/* Metric 1 */}\\n <div style={{ flex: \'0 0 120px\', textAlign: \'center\' }}>\\n <div style={{ fontSize: \'0.75rem\', marginBottom: \'5px\', color: \'#17a2b8\', fontWeight: \'bold\' }}>Metric 1</div>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'#17a2b8\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n color: \'white\',\\n fontSize: \'1.2rem\'\\n }}>\ud83d\udce1</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\', fontWeight: \'bold\', color: \'#17a2b8\' }}>Block P2P<br/>Metric</div>\\n </div>\\n \\n {/* Processing */}\\n <div style={{ flex: \'0 0 100px\', textAlign: \'center\' }}>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'#ffc107\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n fontSize: \'1.2rem\'\\n }}>\u2699\ufe0f</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\' }}>Execute &<br/>Propagate</div>\\n </div>\\n \\n {/* Metric 2 */}\\n <div style={{ flex: \'0 0 120px\', textAlign: \'center\' }}>\\n <div style={{ fontSize: \'0.75rem\', marginBottom: \'5px\', color: \'#28a745\', fontWeight: \'bold\' }}>Metric 2</div>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'#28a745\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n color: \'white\',\\n fontSize: \'1.2rem\'\\n }}>\u2713</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\', fontWeight: \'bold\', color: \'#28a745\' }}>New Head<br/>Metric</div>\\n </div>\\n \\n {/* Deadline */}\\n <div style={{ flex: \'0 0 80px\', textAlign: \'center\' }}>\\n <div style={{\\n width: \'40px\',\\n height: \'40px\',\\n borderRadius: \'50%\',\\n background: \'#dc3545\',\\n margin: \'0 auto\',\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n color: \'white\',\\n fontWeight: \'bold\',\\n fontSize: \'0.9rem\'\\n }}>4s</div>\\n <div style={{ marginTop: \'10px\', fontSize: \'0.75rem\'}}>Attestation Deadline</div>\\n </div>\\n </div>\\n </div>\\n </div>\\n \\n {/* Mobile Timeline - Simple vertical list */}
1\\n <div className=\\"timeline-mobile\\" style={{ display: \'none\', padding: \'0 20px\' }}>\\n <div style={{ display: \'flex\', flexDirection: \'column\', gap: \'20px\' }}>\\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid var(--ifm-color-primary)\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\' }}>\u23f0 Time: ?</div>\\n <div>Block Published</div>\\n </div>\\n\\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid #17a2b8\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\', color: \'#17a2b8\' }}>\ud83d\udce1 Metric 1: Block P2P</div>\\n <div>Nodes receive block via gossip network</div>\\n </div>\\n \\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid #ffc107\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\' }}>\u2699\ufe0f Processing</div>\\n <div>Execute block & propagate blobs</div>\\n </div>\\n \\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid #28a745\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\', color: \'#28a745\' }}>\u2713 Metric 2: New Head</div>\\n <div>Block becomes new chain head</div>\\n </div>\\n \\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'15px\',\\n borderRadius: \'8px\',\\n borderLeft: \'4px solid #dc3545\'\\n }}>\\n <div style={{ fontWeight: \'bold\', marginBottom: \'5px\', color: \'#dc3545\' }}>\u26a0\ufe0f 4s Deadline</div>\\n <div>Attestation deadline</div>\\n </div>\\n </div>\\n </div>\\n \\n {/* Metric descriptions */}\\n <div style={{ display: \'grid\', gridTemplateColumns: \'repeat(auto-fit, minmax(280px, 1fr))\', gap: \'20px\', marginTop: \'50px\' }}>\\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'20px\', \\n borderRadius: \'8px\',\\n border: \'2px solid #17a2b8\',\\n boxShadow: \'0 2px 4px rgba(0,0,0,0.05)\'\\n }}>\\n <h5 style={{ marginTop: 0, color: \'#17a2b8\', fontSize: \'1rem\' }}>\ud83d\udce1 Block P2P</h5>\\n <p style={{ fontSize: \'0.85rem\', marginBottom: \'10px\' }}>When nodes first see the block via gossip network</p>\\n <ul style={{ marginBottom: 0, fontSize: \'0.8rem\', paddingLeft: \'20px\' }}>\\n <li>Measured via <code style={{ fontSize: \'0.75rem\' }}>block_gossip</code> event</li>\\n <li>Occurs <em>before</em> execution</li>\\n <li>Pure network propagation metric</li>\\n </ul>\\n </div>\\n\\n <div style={{ \\n background: \'var(--ifm-background-color)\', \\n padding: \'20px\', \\n borderRadius: \'8px\',\\n border: \'2px solid #28a745\',\\n boxShadow: \'0 2px 4px rgba(0,0,0,0.05)\'\\n }}>\\n <h5 style={{ marginTop: 0, color: \'#28a745\', fontSize: \'1rem\' }}>\u2713 New Head</h5>\\n <p style={{ fontSize: \'0.85rem\', marginBottom: \'10px\' }}>When client considers block as the new chain head</p>\\n <ul style={{ marginBottom: 0, fontSize: \'0.8rem\', paddingLeft: \'20px\' }}>\\n <li>Formula: <code style={{ fontSize: \'0.75rem\' }}>max(block, head, blobs)</code></li>\\n <li>Includes execution time</li>\\n <li>Must wait for all blobs</li>\\n </ul>\\n </div>\\n </div>\\n</div>\\n\\n## Analysis\\n\\n### Is 60M safe?\\n\\n#### 1. Are blocks still being proposed on time?\\nWe can observe this by looking at the P90(min per slot) grouped by gas bucket of the <span style={{ color: \'#17a2b8\', fontWeight: \'bold\' }}>\ud83d\udce1 P2P metric.</span> This is a fancy way of saying:\\n- First find the earliest time any Xatu node saw the block for the slot\\n- Then for each gas bucket, find the 90th percentile of these first seen times\\n\\nThis metric gives us insight in to block proposers ability to build and propagate blocks on time. If even the fastest nodes are receiving blocks significantly later as gas usage increases, it suggests proposers may be struggling when building and propagating larger blocks.\\n\\n<ClickableImage\\n src=\\"/img/blog/60m-gas-testnets/sepolia-gossip.png\\"\\n alt=\\"Sepolia\\"\\n caption=\\"Figure 1: Sepolia P90(min per slot) grouped by gas bucket of the \ud83d\udce1 P2P metric. At 60M gas usage, 90% of blocks are first seen by 680ms.\\"\\n/>\\n\\n<ClickableImage\\n src=\\"/img/blog/60m-gas-testnets/hoodi-gossip.png\\"\\n alt=\\"Hoodi\\"\\n caption=\\"Figure 2: Hoodi P90(min per slot) grouped by gas bucket of the \ud83d\udce1 P2P metric. At 60M gas usage, 90% of blocks are first seen by 1016ms.\\"\\n/>\\n\\nAcross both networks we can see that blocks are continuing to be proposed on time. As expected, we observe a slight increase in the block proposal time as the gas usage of the blocks increases.\\n\\n<Alert type=\\"success\\">\\n <strong>Blocks are being proposed on time.</strong>\\n</Alert>\\n\\n#### 2. Are blocks being accepted before 4s?\\nWe can observe this by looking at the 90th percentile of our <span style={{ color: \'#28a745\', fontWeight: \'bold\' }}>\u2713 New Head metric</span>. Using the 90th percentile allows us to filter out any outliers. As a reminder, 66% of the network needs to see the block and any associated blobs within 4s for it to be considered valid.\\n\\n<ClickableImage\\n src=\\"/img/blog/60m-gas-testnets/sepolia-head.png\\"\\n alt=\\"Sepolia\\"\\n caption=\\"Figure 3:
1Sepolia p90(\u2713 New Head metric). Blocks are being accepted before 4s.\\"\\n/>\\n\\n<ClickableImage\\n src=\\"/img/blog/60m-gas-testnets/hoodi-head.png\\"\\n alt=\\"Hoodi\\"\\n caption=\\"Figure 4: Hoodi p90(\u2713 New Head metric). Blocks are being accepted before 4s.\\"\\n/>\\n\\nAgain, we can see that blocks are being accepted well before the 4s. As expected, we observe a similar increase in this metric as the gas usage of the blocks increases. It\'s worth noting here that Hoodi has a smaller relative arrival time than Sepolia as gas usage increases.\\n\\n<Alert type=\\"success\\">\\n <strong>Blocks are being accepted well-before 4s.</strong>\\n</Alert>\\n\\n### Beyond 60M?\\n\\nLet\'s venture into mad scientist territory: what happens if we go beyond 60M on these networks? We can make a few assumptions and extrapolate the current performance out to see when we would cross the 4s threshold. In reality we would expect to see hidden bottlenecks and other issues pop up, but it\'s still an interesting exercise.\\n\\n<ClickableImage\\n src=\\"/img/blog/60m-gas-testnets/sepolia-extrapolation.png\\"\\n alt=\\"Sepolia\\"\\n caption=\\"Figure 5: Sepolia p90(both metrics) with trend lines extrapolated out to cross the 4s threshold.\\"\\n/>\\n\\nThe <span style={{ color: \'#28a745\', fontWeight: \'bold\' }}>\u2713 New Head metric</span> crosses at 151M gas, and the <span style={{ color: \'#17a2b8\', fontWeight: \'bold\' }}>\ud83d\udce1 P2P metric</span> crosses at 500M gas. This disconnect between the two metrics shows the overhead the network pays from _executing_ the block as well as propagating any blobs at the same time.\\n\\n<ClickableImage\\n src=\\"/img/blog/60m-gas-testnets/hoodi-extrapolation.png\\"\\n alt=\\"Hoodi\\"\\n caption=\\"Figure 6: Hoodi p90(both metrics) with trend lines extrapolated out to cross the 4s threshold.\\"\\n/>\\n\\nEven more interestingly, on Hoodi, we see more extreme results, with the <span style={{ color: \'#28a745\', fontWeight: \'bold\' }}>\u2713 New Head metric</span> crossing at 832M gas (5x Sepolia), and the <span style={{ color: \'#17a2b8\', fontWeight: \'bold\' }}>\ud83d\udce1 P2P metric</span> crossing at 1,994M gas (4x Sepolia). There is a large difference between the two networks on both metrics. This may indicate a sensitivity to large execution state size. **This is a particularly interesting result as Mainnet has a much larger execution state size and we\'ll be monitoring this closely as we continue to scale.**\\n\\n\\n## Acknowledgments\\n\\nSpecial thanks to our contributoor community for providing the distributed monitoring data that makes this analysis possible. Their participation gives us unprecedented visibility into Ethereum network behavior.\\n\\n_Want to contribute to our monitoring efforts? [Become a contributoor!](/contribute-data/)_\\n\\n## Conclusion\\n\\nBased on the data from Hoodi & Sepolia, 60M is safe as far as block/blob propagation is concerned. It\'s **very** important to note that these testnets are not representative of Mainnet. We\'ll be conducting additional analysis on Mainnet in the coming days, but for now we can say that 60M is possible on a fundamental level.\\n\\n<div style={{ textAlign: \'center\', marginTop: \'40px\', fontSize: \'1.5rem\', fontWeight: \'bold\' }}>\\n \ud83d\ude80\ud83d\ude80\ud83d\ude80\\n</div>"},{"id":"pectra-mainnet-checklist","metadata":{"permalink":"/posts/pectra-mainnet-checklist","source":"@site/blog/2025-05-01-pectra-mainnet/index.md","title":"Pectra Mainnet Checklist","description":"Prepare for the Pectra Mainnet launch!","date":"2025-05-01T00:00:00.000Z","tags":[{"inline":true,"label":"mainnet","permalink":"/posts/tags/mainnet"},{"inline":true,"label":"pectra","permalink":"/posts/tags/pectra"},{"inline":true,"label":"upgrade","permalink":"/posts/tags/upgrade"},{"inline":true,"label":"electra","permalink":"/posts/tags/electra"},{"inline":true,"label":"prague","permalink":"/posts/tags/prague"}],"readingTime":17.64,"hasTruncateMarker":false,"authors":[{"name":"ethPandaOps Team","title":"ethPandaOps Team","url":"https://github.com/ethpandaops","imageURL":"https://github.com/ethpandaops.png","key":"team","page":null}],"frontMatter":{"slug":"pectra-mainnet-checklist","title":"Pectra Mainnet Checklist","authors":["team"],"description":"Prepare for the Pectra Mainnet launch!","tags":["mainnet","pectra","upgrade","electra","prague"],"image":"/img/blog/pectra-mainnet.jpg","relatedLinks":[{"name":"Ethereum Foundation Blog Post","url":"https://blog.ethereum.org/2025/04/23/pectra-mainnet"},{"name":"Upgrade Livestream","url":"https://www.youtube.com/watch?v=NbxaCZYtW-I"}]},"unlisted":false,"prevItem":{"title":"60M Gas Limit on Sepolia & Hoodi","permalink":"/posts/60m-gas-sepolia-hoodi"},"nextItem":{"title":"Electra on Hoodi: Attestation Packing","permalink":"/posts/hoodi-attestation-packing"}}
1,"content":"import { useState, useEffect } from \'react\';\\n\\nexport const PectraUpgradeWizard = () => {\\n // Client data\\n const clientVersions = {\\n el: {\\n geth: { name: \\"Geth\\", version: \\"v1.15.9\\", url: \\"https://github.com/ethereum/go-ethereum\\" },\\n nethermind: { name: \\"Nethermind\\", version: \\"v1.31.9\\", url: \\"https://github.com/NethermindEth/nethermind\\" },\\n besu: { name: \\"Besu\\", version: \\"v25.4.1\\", url: \\"https://github.com/besu-eth/besu\\" },\\n erigon: { name: \\"Erigon\\", version: \\"v3.0.2\\", url: \\"https://github.com/ledgerwatch/erigon\\" },\\n reth: { name: \\"Reth\\", version: \\"v1.3.12\\", url: \\"https://github.com/paradigmxyz/reth\\" }\\n },\\n cl: {\\n lighthouse: { name: \\"Lighthouse\\", version: \\"v7.0.1\\", url: \\"https://github.com/sigp/lighthouse\\" },\\n nimbus: { name: \\"Nimbus\\", version: \\"v25.4.1\\", url: \\"https://github.com/status-im/nimbus-eth2\\" },\\n prysm: { name: \\"Prysm\\", version: \\"v6.0.1\\", url: \\"https://github.com/prysmaticlabs/prysm\\" },\\n teku: { name: \\"Teku\\", version: \\"v25.4.1\\", url: \\"https://github.com/ConsenSys/teku\\" },\\n lodestar: { name: \\"Lodestar\\", version: \\"v1.29.0\\", url: \\"https://github.com/ChainSafe/lodestar\\" },\\n grandine: { name: \\"Grandine\\", version: \\"v1.1.0\\", url: \\"https://github.com/grandinetech/grandine\\" }\\n },\\n mev: { name: \\"MEV-Boost\\", version: \\"v1.9\\", url: \\"https://github.com/flashbots/mev-boost\\" }\\n };\\n \\n // State hooks\\n const [step, setStep] = useState(\'start\');\\n const [isNode, setIsNode] = useState(false);\\n const [isValidator, setIsValidator] = useState(false);\\n const [isMevBoost, setIsMevBoost] = useState(false);\\n const [elClient, setElClient] = useState(\'\');\\n const [clClient, setClClient] = useState(\'\');\\n const [elUpgraded, setElUpgraded] = useState(false);\\n const [clUpgraded, setClUpgraded] = useState(false);\\n const [validatorUpgraded, setValidatorUpgraded] = useState(false);\\n const [mevUpgraded, setMevUpgraded] = useState(false);\\n const [savedSelections, setSavedSelections] = useState(false);\\n \\n // Load state from localStorage on component mount\\n useEffect(() => {\\n const savedSetup = localStorage.getItem(\'pectraUserSetup\');\\n if (savedSetup) {\\n try {\\n const parsed = JSON.parse(savedSetup);\\n setIsNode(parsed.isNode || false);\\n setIsValidator(parsed.isValidator || false);\\n setIsMevBoost(parsed.isMevBoost || false);\\n setElClient(parsed.elClient || \'\');\\n setClClient(parsed.clClient || \'\');\\n setElUpgraded(parsed.elUpgraded || false);\\n setClUpgraded(parsed.clUpgraded || false);\\n setValidatorUpgraded(parsed.validatorUpgraded || false);\\n setMevUpgraded(parsed.mevUpgraded || false);\\n setSavedSelections(true);\\n \\n // Start at the appropriate step based on saved data\\n if (parsed.isNode) {\\n if (!parsed.elClient) {\\n setStep(\'el_client\');\\n } else if (!parsed.elUpgraded) {\\n setStep(\'el_upgraded\');\\n } else if (!parsed.clClient) {\\n setStep(\'cl_client\');\\n } else if (!parsed.clUpgraded) {\\n setStep(\'cl_upgraded\');\\n } else if (parsed.isValidator && !parsed.validatorUpgraded) {\\n setStep(\'validator_upgraded\');\\n } else if (parsed.isMevBoost && !parsed.mevUpgraded) {\\n setStep(\'mev_upgraded\');\\n } else {\\n setStep(\'complete\');\\n }\\n }\\n } catch (e) {\\n console.error(\\"Error parsing saved setup\\", e);\\n }\\n }\\n }, []);\\n \\n // Save to localStorage whenever state changes\\n useEffect(() => {\\n if (savedSelections) {\\n localStorage.setItem(\'pectraUserSetup\', JSON.stringify({\\n isNode,\\n isValidator,\\n isMevBoost,\\n elClient,\\n clClient,\\n elUpgraded,\\n clUpgraded,\\n validatorUpgraded,\\n mevUpgraded\\n }));\\n }\\n }, [isNode, isValidator, isMevBoost, elClient, clClient, elUpgraded, clUpgraded, validatorUpgraded, mevUpgraded, savedSelections]);\\n \\n // Check if all necessary upgrades are complete\\n const allUpgradesComplete = () => {\\n if (isNode) {\\n if (!elUpgraded || !clUpgraded) return false;\\n if (isValidator && !validatorUpgraded) return false;\\n if (isMevBoost && !mevUpgraded) return false;\\n }\\n return true;\\n };\\n \\n // Update state and move to next step\\n const handleAnswer = (answer, nextStep) =>
1 {\\n if (!savedSelections) setSavedSelections(true);\\n \\n // Update state based on the question\\n switch (step) {\\n case \'start\':\\n setIsNode(answer === \'yes\');\\n break;\\n case \'el_client\':\\n setElClient(answer);\\n break;\\n case \'el_upgraded\':\\n setElUpgraded(answer === \'yes\');\\n break;\\n case \'cl_client\':\\n setClClient(answer);\\n break;\\n case \'cl_upgraded\':\\n setClUpgraded(answer === \'yes\');\\n break;\\n case \'validator\':\\n setIsValidator(answer === \'yes\');\\n break;\\n case \'validator_upgraded\':\\n setValidatorUpgraded(answer === \'yes\');\\n break;\\n case \'mev\':\\n setIsMevBoost(answer === \'yes\');\\n break;\\n case \'mev_upgraded\':\\n setMevUpgraded(answer === \'yes\');\\n break;\\n }\\n \\n // Move to the next step\\n setStep(nextStep);\\n };\\n \\n // Determine the next step based on current state\\n const getNextStep = (currentStep, answer) => {\\n switch (currentStep) {\\n case \'start\':\\n return answer === \'yes\' ? \'el_client\' : \'not_running_node\';\\n case \'el_client\':\\n return \'el_upgraded\';\\n case \'el_upgraded\':\\n return \'cl_client\';\\n case \'cl_client\':\\n return \'cl_upgraded\';\\n case \'cl_upgraded\':\\n return \'validator\';\\n case \'validator\':\\n return answer === \'yes\' ? \'validator_upgraded\' : \'complete\';\\n case \'validator_upgraded\':\\n return \'mev\';\\n case \'mev\':\\n return answer === \'yes\' ? \'mev_upgraded\' : \'complete\';\\n case \'mev_upgraded\':\\n return \'complete\';\\n default:\\n return \'complete\';\\n }\\n };\\n \\n // Get the text for the current step\\n const getStepText = () => {\\n switch (step) {\\n case \'start\':\\n return \'Are you running an Ethereum node?\';\\n case \'el_client\':\\n return \'Which execution client are you using?\';\\n case \'el_upgraded\':\\n return `Have you upgraded your ${elClient ? elClient.charAt(0).toUpperCase() + elClient.slice(1) : \'execution client\'} to at least ${elClient && clientVersions.el[elClient] ? clientVersions.el[elClient].version : \'the latest version\'}?`;\\n case \'cl_client\':\\n return \'Which consensus client are you using?\';\\n case \'cl_upgraded\':\\n return `Have you upgraded your ${clClient ? clClient.charAt(0).toUpperCase() + clClient.slice(1) : \'consensus client\'} to at least ${clClient && clientVersions.cl[clClient] ? clientVersions.cl[clClient].version : \'the latest version\'}?`;\\n case \'validator\':\\n return \'Are you running a validator?\';\\n case \'validator_upgraded\':\\n return `Have you upgraded your ${clClient ? clClient.charAt(0).toUpperCase() + clClient.slice(1) : \'consensus client\'} validator client to at least ${clClient && clientVersions.cl[clClient] ? clientVersions.cl[clClient].version : \'the latest version\'}?`;\\n case \'mev\':\\n return \'Are you running MEV-Boost?\';\\n case \'mev_upgraded\':\\n return `Have you upgraded MEV-Boost to at least ${clientVersions.mev.version}?`;\\n case \'not_running_node\':\\n case \'complete\':\\n return allUpgradesComplete() \\n ? \'Congratulations! You\\\\\'re all set for the Pectra upgrade! \ud83c\udf89\' \\n : \'Please complete the necessary upgrades before the Pectra fork.\';\\n default:\\n return \'Let\\\\\'s check your readiness for the Pectra upgrade\';\\n }\\n };\\n \\n // Render the current step\\n const renderStep = () => {\\n // Non-question steps\\n if (step === \'not_running_node\') {\\n return (\\n <div style={{ textAlign: \'center\', margin: \'24px 0\' }}>\\n <div style={{ fontSize: \'64px\', marginBottom: \'16px\' }}>\ud83c\udf89</div>\\n <h3 style={{ color: \'var(--ifm-color-success)\' }}>{getStepText()}</h3>\\n <p>You don\'t need to do anything for this upgrade. You can still keep an eye on the Pectra upgrade through the monitoring resources below.</p>\\n <button\\n onClick={() => setStep(\'start\')}\\n style={{\\n marginTop: \'16px\',\\n padding: \'8px 16px\',\\n backgroundColor: \'var(--ifm-color-emphasis-200)\',\\n color: \'var(--ifm-color-emphasis-700)\',\\n border: \'none\',\\n borderRadius: \'4px\',\\n cursor: \'pointer\'\\n }}\\n >\\n Start Again\\n </button>\\n </div>\\n );\\n }\\n \\n if (step === \'complete\') {\\n return (\\n <div style={{ textAlign: \'center\', margin: \'24px 0\' }}>\\n {allUpgradesComplete() ? (\\n <>\\n <div style={{ fontSize: \'64px\', marginBottom: \'16px\' }}>\ud83c\udf89</div>\\n <h3 style={{ color: \'var(--ifm-color-success)\' }}>{getStepText()}</h3>\\n <p>Your node is ready for the upcoming Pectra upgrade.</p>\\n \\n {isNode && (\\n <div style={{\\n margin: \'24px auto\',\\n padding: \'16px\',\\n backgroundColor: \'var(--ifm-color-info-contrast-background)\',\\n border: \'1px solid var(--ifm-color-emphasis-300)\',\\n borderRadius: \'8px\',\\n textAlign: \'left\'\\n }}>\\n <h4 style={{ marginTop: \'0\' }}>
1\\n <span style={{ marginRight: \'8px\' }}>\ud83d\udcca</span>\\n Contribute to ethPandaOps Research\\n </h4>\\n <p>\\n Since you\'re running a node, would you like to contribute your node\'s data to help the public Xatu dataset? Data is published openly in a privacy-focused manner and used for research and analysis.\\n </p>\\n <a \\n href=\\"https://docs.google.com/forms/u/3/d/e/1FAIpQLSfSCHLZv2xdDKceQ3ajzYbkm2jkQnG0lVBhjEX4c1Jxvf8QfA/viewform\\"\\n target=\\"_blank\\"\\n rel=\\"noopener noreferrer\\"\\n style={{\\n display: \'inline-block\',\\n marginTop: \'8px\',\\n padding: \'8px 16px\',\\n backgroundColor: \'var(--ifm-color-info)\',\\n color: \'white\',\\n borderRadius: \'4px\',\\n textDecoration: \'none\',\\n fontWeight: \'bold\'\\n }}\\n >\\n Sign Up to Contribute Data\\n </a>\\n </div>\\n )}\\n </>\\n ) : (\\n <>\\n <div style={{ fontSize: \'64px\', marginBottom: \'16px\' }}>\u26a0\ufe0f</div>\\n <h3 style={{ color: \'var(--ifm-color-warning)\' }}>{getStepText()}</h3>\\n <p>Please make sure to upgrade all components before the fork.</p>\\n \\n {isNode && (\\n <div style={{\\n margin: \'24px auto\',\\n padding: \'16px\',\\n maxWidth: \'600px\',\\n backgroundColor: \'var(--ifm-color-info-contrast-background)\',\\n border: \'1px solid var(--ifm-color-emphasis-300)\',\\n borderRadius: \'8px\',\\n textAlign: \'left\'\\n }}>\\n <h4 style={{ color: \'var(--ifm-color-info)\', marginTop: \'0\' }}>\\n <span style={{ marginRight: \'8px\' }}>\ud83d\udcca</span>\\n Contribute to ethPandaOps Research\\n </h4>\\n <p>\\n Since you\'re running a node, would you like to contribute your node data to help the ethPandaOps team analyze network health and performance?\\n </p>\\n <p>\\n Your contribution helps improve Ethereum for everyone by providing valuable insights into the network\'s behavior.\\n </p>\\n <a \\n href=\\"https://docs.google.com/forms/u/3/d/e/1FAIpQLSfSCHLZv2xdDKceQ3ajzYbkm2jkQnG0lVBhjEX4c1Jxvf8QfA/viewform\\"\\n target=\\"_blank\\"\\n rel=\\"noopener noreferrer\\"\\n style={{\\n display: \'inline-block\',\\n marginTop: \'8px\',\\n padding: \'8px 16px\',\\n backgroundColor: \'var(--ifm-color-info)\',\\n color: \'white\',\\n borderRadius: \'4px\',\\n textDecoration: \'none\',\\n fontWeight: \'bold\'\\n }}\\n >\\n Sign Up to Contribute Data\\n </a>\\n </div>\\n )}\\n </>\\n )}\\n <button\\n onClick={() => setStep(\'start\')}\\n style={{\\n marginTop: \'16px\',\\n padding: \'8px 16px\',\\n backgroundColor: \'var(--ifm-color-emphasis-200)\',\\n color: \'var(--ifm-color-emphasis-700)\',\\n border: \'none\',\\n borderRadius: \'4px\',\\n cursor: \'pointer\'\\n }}\\n >\\n Restart Checklist\\n </button>\\n </div>\\n );\\n }\\n \\n // Questions with Yes/No answers\\n if ([\'start\', \'el_upgraded\', \'cl_upgraded\', \'validator\', \'validator_upgraded\', \'mev\', \'mev_upgraded\'].includes(step)) {\\n // Check if this is an upgrade-related step\\n const isUpgradeStep = [\'el_upgraded\', \'cl_upgraded\', \'validator_upgraded\', \'mev_upgraded\'].includes(step);\\n \\n return (\\n <div style={{ margin: \'24px 0\', textAlign: \'center\' }}>\\n <h3>{getStepText()}</h3>\\n \\n {isUpgradeStep && (\\n <div style={{ \\n marginTop: \'12px\', \\n padding: \'8px 16px\',\\n borderRadius: \'4px\',\\n backgroundColor: \'var(--ifm-color-warning-lightest)\',\\n border: \'1px solid var(--ifm-color-warning-light)\',\\n color: \'var(--ifm-color-warning-darkest)\',\\n fontSize: \'0.9rem\'\\n }}>\\n \u26a0\ufe0f You must upgrade your client to continue running on the network after the Pectra fork.\\n </div>\\n )}\\n \\n <div style={{ \\n display: \'flex\', \\n justifyContent: \'space-around\',\\n marginTop: \'24px\'\\n }}>\\n <button\\n onClick={() => handleAnswer(\'yes\', getNextStep(step, \'yes\'))}\\n style={{\\n padding: \'12px 40px\',\\n backgroundColor: \'var(--ifm-color-success)\',\\n color: \'white\',\\n border: \'none\',\\n borderRadius: \'4px\',\\n cursor: \'pointer\',\\n fontWeight: \'bold\',\\n fontSize: \'16px\'\\n }}\\n >\\n Yes\\n </button>\\n <button\\n onClick={() => isUpgradeStep ? null : handleAnswer(\'no\', getNextStep(step, \'no\'))}\\n style={{\\n padding: \'12px 40px\',\\n backgroundColor: isUpgradeStep ? \'var(--ifm-color-gray-400)\' : \'var(--ifm-color-danger)\',\\n color: \'white\',\\n border: \'none\',\\n borderRadius: \'4px\',\\n cursor: isUpgradeStep ? \'not-allowed\' : \'pointer\',\\n fontWeight: \'bold\',\\n fontSize: \'16px\',\\n opacity: isUpgradeStep ? 0.7 : 1\\n }}\\n disabled={isUpgradeStep}\\n title={isUpgradeStep ? \\"You must upgrade before the fork\\" : \\"\\"}\\n >\\n No\\n </button>\\n </div>\\n </div>\\n );\\n }\\n \\n // Client selection steps\\n if ([\'el_client\', \'cl_client\'].includes(step)) {\\n const clients = step === \'el_client\' \\n ? [\\n { value: \'geth\', label: \'Geth\' },\\n { value: \'nethermind\', label: \'Nethermind\' },\\n { value: \'besu\', label: \'Besu\' },\\n { value: \'erigon\', label: \'Erigon\' },\\n { value: \'reth\', label: \'Reth\' }\\n ]\\n : [\\n { value: \'lighthouse\', label: \'Lighthouse\' },\\n { value: \'nimbus\', label: \'Nimbus\' },\\n { value: \'prysm\', label: \'Prysm\' },\\n { value: \'teku\', label: \'Teku\' },\\n { value: \'lodestar\', label: \'Lodestar\' },\\n { value: \'grandine\', label: \'Grandine\' }\\n ];\\n \\n return (\\n <div style={{ margin: \'24px 0\', textAlign: \'center\' }}>\\n <h3>{getStepText()}</h3>\\n <div style={{ \\n display: \'flex\', \\n flexDirection: \'column\',\\n gap: \'12px\',\\n marginTop: \'24px\'\\n }}>\\n {clients.map(client => (\\n <button\\n key={client.value}\\n onClick={() => handleAnswer(client.value, getNextStep(step))}\\n style={{\\n padding: \'12px 24px\',\\n backgroundColor: \'var(--ifm-color-primary)\',\\n color: \'white\',\\n border: \'none\',\\n borderRadius: \'4px\',\\n cursor: \'pointer\',\\n fontWeight: \'bold\',\\n fontSize: \'16px\'\\n }}\\n >\\n {client.label}\\n </button>\\n ))}\\n </div>\\n </div>\\n );\\n }\\n \\n // Default fallback\\n return <div>Loading...</div>;\\n };\\n \\n // Display progress indicator\\n const renderProgress = () => {\\n // Define all possible steps in order\\n const allSteps = [\'start\', \'el_client\', \'el_upgraded\', \'cl_client\', \'cl_upgraded\', \'validator\', \'validator_upgraded\', \'mev\', \'mev_upgraded\', \'complete\'];\\n \\n // Skip validator steps if not a validator\\n let progressSteps = allSteps;\\n if (step !== \'start\' && !isValidator) {\\n progressSteps = allSteps.filter(s => ![\'validator\', \'validator_upgraded\'].includes(s));\\n }\\n \\n // Calculate progress\\n const currentIndex = progressSteps.indexOf(step);\\n const totalSteps = progressSteps.length;\\n const progress = currentIndex / (totalSteps - 1) * 100;\\n \\n return (\\n <div style={{ marginBottom: \'32px\' }}>\\n <div style={{ \\n width: \'100%\', \\n height: \'8px\', \\n backgroundColor: \'var(--ifm-color-emphasis-200)\',\\n borderRadius: \'4px\',\\n overflow: \'hidden\'\\n }}>\\n <div style={{ \\n width: `${progress}%`, \\n height: \'100%\', \\n backgroundColor: \'var(--ifm-color-primary)\',\\n transition: \'width 0.5s ease\'\\n }} />\\n </div>\\n <div style={{ \\n display: \'flex\', \\n justifyContent: \'space-between\',\\n marginTop: \'4px\',\\n fontSize: \'12px\',\\n color: \'var(--ifm-color-emphasis-600)\'\\n }}>
1\\n <span>Start</span>\\n <span>Progress</span>\\n <span>Complete</span>\\n </div>\\n </div>\\n );\\n };\\n \\n return (\\n <div style={{ \\n marginBottom: \'32px\',\\n padding: \'24px\',\\n borderRadius: \'8px\',\\n backgroundColor: \'var(--ifm-background-surface-color)\',\\n border: \'1px solid var(--ifm-color-emphasis-300)\'\\n }}>\\n {renderProgress()}\\n {renderStep()}\\n </div>\\n );\\n};\\n\\n## It\'s Forking Time!\\n\\nEthereum\'s next network upgrade, Pectra, is scheduled to activate on Mainnet at epoch 364032, which lands at [May 7, 2025, at 10:05:11 UTC](https://blog.ethereum.org/2025/04/23/pectra-mainnet). This is your definitive guide to the Pectra upgrade.\\n\\n\\nexport const PectraCountdown = () => {\\n const [timeLeft, setTimeLeft] = useState({\\n days: 0,\\n hours: 0,\\n minutes: 0,\\n seconds: 0\\n });\\n const [isActivated, setIsActivated] = useState(false);\\n \\n // Activation details\\n const targetDate = new Date(\'2025-05-07T10:05:11Z\');\\n const targetEpoch = 364032;\\n const slotsPerEpoch = 32;\\n const secondsPerSlot = 12;\\n \\n // Add useEffect hook for checking screen width\\n const [isMobile, setIsMobile] = useState(false);\\n \\n useEffect(() => {\\n const calculateTimeLeft = () => {\\n const now = new Date();\\n const difference = targetDate - now;\\n \\n if (difference > 0) {\\n setTimeLeft({\\n days: Math.floor(difference / (1000 * 60 * 60 * 24)),\\n hours: Math.floor((difference / (1000 * 60 * 60)) % 24),\\n minutes: Math.floor((difference / 1000 / 60) % 60),\\n seconds: Math.floor((difference / 1000) % 60)\\n });\\n setIsActivated(false);\\n } else {\\n setTimeLeft({ days: 0, hours: 0, minutes: 0, seconds: 0 });\\n setIsActivated(true);\\n }\\n };\\n \\n calculateTimeLeft();\\n const timer = setInterval(calculateTimeLeft, 1000);\\n \\n return () => clearInterval(timer);\\n }, []);\\n \\n useEffect(() => {\\n const checkIfMobile = () => {\\n setIsMobile(window.innerWidth < 550);\\n };\\n \\n checkIfMobile();\\n window.addEventListener(\'resize\', checkIfMobile);\\n \\n return () => {\\n window.removeEventListener(\'resize\', checkIfMobile);\\n };\\n }, []);\\n \\n // Calculate remaining slots and epochs\\n const totalSecondsRemaining = timeLeft.days * 86400 + timeLeft.hours * 3600 + timeLeft.minutes * 60 + timeLeft.seconds;\\n const remainingSlots = Math.ceil(totalSecondsRemaining / secondsPerSlot);\\n const remainingEpochs = Math.ceil(remainingSlots / slotsPerEpoch);\\n \\n // Format date for consistent display with mobile friendliness\\n const formatDate = (date) => {\\n if (isMobile) {\\n return date.toLocaleString(undefined, {\\n year: \'numeric\',\\n month: \'short\',\\n day: \'numeric\',\\n hour: \'2-digit\',\\n minute: \'2-digit\',\\n timeZoneName: \'short\'\\n });\\n }\\n \\n return date.toLocaleString(undefined, {\\n year: \'numeric\',\\n month: \'long\',\\n day: \'numeric\',\\n hour: \'2-digit\',\\n minute: \'2-digit\',\\n second: \'2-digit\',\\n timeZoneName: \'short\'\\n });\\n };\\n \\n if (isActivated) {\\n return (\\n <div style={{\\n width: \'100%\',\\n display: \'flex\',\\n flexDirection: \'column\',\\n alignItems: \'center\',\\n backgroundColor: \'var(--ifm-color-success-lightest)\',\\n borderRadius: \'12px\',\\n padding: \'12px\',\\n marginTop: \'12px\',\\n marginBottom: \'12px\',\\n textAlign: \'center\',\\n boxShadow: \'0 4px 8px rgba(0, 0, 0, 0.05)\'\\n }}>\\n <div style={{ fontSize: \'72px\', marginBottom: \'8px\' }}>\ud83c\udf89</div>\\n <h2 style={{ color: \'var(--ifm-color-success-darkest)\', marginBottom: \'16px\' }}>\\n Pectra is Active!\\n </h2>\\n <p style={{ fontSize: \'18px\', maxWidth: \'700px\' }}>\\n The Ethereum network upgraded to Pectra at epoch {targetEpoch}.\\n Congratulations to all who contributed to this milestone!\\n </p>\\n <a \\n href={`https://beaconcha.in/epoch/${targetEpoch}`}\\n target=\\"_blank\\"\\n rel=\\"noopener noreferrer\\"\\n style={{\\n display: \'inline-block\',\\n marginTop: \'16px\',\\n padding: \'8px 16px\',\\n backgroundColor: \'var(--ifm-color-success)\',\\n color: \'white\',\\n borderRadius: \'4px\',\\n textDecoration: \'none\',\\n fontWeight: \'bold\'\\n }}\\n >\\n View Activation Data\\n </a>\\n </div>\\n );\\n }\\n \\n return (\\n <div style={{ \\n display: \'flex\', \\n flexDirection: \'column\',\\n alignItems: \'center\', \\n marginBottom: \'32px\',\\n marginTop: \'24px\',\\n width: \'100%\',\\n position: \'relative\',\\n overflow: \'hidden\'\\n }}>
1\\n {/* Ethereum Logo Background */}\\n <div style={{\\n position: \'absolute\',\\n top: \'50%\',\\n right: \'5%\',\\n transform: \'translateY(-50%)\',\\n opacity: 0.05,\\n zIndex: 0,\\n display: \'flex\',\\n justifyContent: \'center\',\\n alignItems: \'center\'\\n }}>\\n <div style={{ \\n width: \'200px\',\\n height: \'320px\',\\n background: \'var(--ifm-color-primary)\',\\n clipPath: \'polygon(50% 0%, 100% 25%, 100% 75%, 50% 100%, 0% 75%, 0% 25%)\',\\n }}></div>\\n </div>\\n \\n {/* Main Countdown Display */}\\n <div style={{ \\n display: \'flex\', \\n flexWrap: isMobile ? \'nowrap\' : \'wrap\',\\n gap: isMobile ? \'8px\' : \'16px\', \\n textAlign: \'center\',\\n backgroundColor: \'var(--ifm-background-surface-color)\',\\n padding: isMobile ? \'12px 8px\' : \'24px 16px\',\\n borderRadius: \'12px\',\\n boxShadow: \'0 8px 24px rgba(0, 0, 0, 0.07)\',\\n marginBottom: \'20px\',\\n border: \'1px solid var(--ifm-color-emphasis-200)\',\\n width: \'100%\',\\n maxWidth: \'600px\',\\n justifyContent: \'center\',\\n position: \'relative\',\\n zIndex: 1\\n }}>\\n {[\\n { value: timeLeft.days, label: \'DAYS\', shortLabel: \'DAY\' },\\n { value: timeLeft.hours, label: \'HOURS\', shortLabel: \'HOUR\' },\\n { value: timeLeft.minutes, label: \'MINUTES\', shortLabel: \'MIN\' },\\n { value: timeLeft.seconds, label: \'SECONDS\', shortLabel: \'SEC\' }\\n ].map(({ value, label, shortLabel }) => (\\n <div key={label} style={{ \\n width: isMobile ? \'25%\' : \'auto\',\\n minWidth: isMobile ? \'20%\' : \'75px\',\\n display: \'flex\',\\n flexDirection: \'column\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n margin: \'0\',\\n padding: isMobile ? \'0 2px\' : \'0\'\\n }}>\\n <div style={{ \\n fontSize: isMobile ? \'20px\' : \'48px\', \\n fontWeight: \'700\', \\n background: \'linear-gradient(90deg, #0969da 0%, #9c44dc 100%)\',\\n WebkitBackgroundClip: \'text\',\\n WebkitTextFillColor: \'transparent\',\\n backgroundClip: \'text\',\\n textFillColor: \'transparent\',\\n lineHeight: \'1.1\',\\n padding: isMobile ? \'2px 0\' : \'4px 0\'\\n }}>\\n {String(value).padStart(2, \'0\')}\\n </div>\\n <div style={{ \\n fontSize: isMobile ? \'7px\' : \'14px\', \\n textTransform: \'uppercase\', \\n letterSpacing: isMobile ? \'0\' : \'1px\',\\n fontWeight: \'600\',\\n color: \'var(--ifm-color-emphasis-700)\'\\n }}>\\n {isMobile ? shortLabel : label}\\n </div>\\n </div>\\n ))}\\n </div>\\n \\n {/* Activation Time Display */}\\n <div style={{\\n display: \'flex\',\\n alignItems: \'center\',\\n justifyContent: \'center\',\\n padding: isMobile ? \'8px 12px\' : \'10px 16px\',\\n borderRadius: \'8px\',\\n backgroundColor: \'var(--ifm-color-emphasis-100)\',\\n boxShadow: \'0 2px 6px rgba(0, 0, 0, 0.04)\',\\n fontSize: isMobile ? \'0.8rem\' : \'0.95rem\',\\n position: \'relative\',\\n zIndex: 1,\\n width: \'100%\',\\n maxWidth: \'600px\',\\n textAlign: \'center\'\\n }}>\\n <div style={{ \\n display: \'flex\',\\n flexDirection: isMobile ? \'column\' : \'row\',\\n alignItems: \'center\',\\n gap: \'4px\'\\n }}>\\n <span style={{ \\n marginRight: isMobile ? \'0\' : \'8px\',\\n fontSize: \'16px\'\\n }}>\\n \u23f0\\n </span>\\n <div>\\n {!isMobile && <strong>Local Activation Time:</strong>} {formatDate(targetDate)}\\n </div>\\n </div>\\n </div>\\n </div>\\n );\\n};\\n\\n<PectraCountdown />\\n\\n### Pectra?\\nPectra is the amalgamation of the execution layer\'s \\"Prague\\" upgrade and consensus layer\'s \\"Electra\\" upgrade. It introduces enhancements aimed at improving scalability, validator operations, and user experience, and is one of the most significant upgrades to Ethereum to date. For more information on the upgrade, check out the [Ethereum Foundation blog post](https://blog.ethereum.org/2025/04/23/pectra-mainnet).\\n\\n## Upgrade Checklist\\nPectra is a hard fork for the Ethereum network, meaning that you\'ll need to upgrade your node to participate. We\'ve created a simple tool to help you through the process.\\n<PectraUpgradeWizard />\\n\\n## ethPandaOps Tooling\\n\\nOur tools are all ready for the upgrade. If you\'re running any of these, you\'ll need to upgrade to the latest versions to ensure compatibility.\\n\\nexport const ToolingTable = () => {\\n const [isMobile, setIsMobile] = useState(false);\\n \\n useEffect(() => {\\n const checkIfMobile = () => {\\n setIsMobile(window.innerWidth < 768);\\n };\\n \\n checkIfMobile();\\n window.addEventListener(\'resize\', checkIfMobile);\\n \\n return () => {\\n window.removeEventListener(\'resize\', checkIfMobile);\\n };\\n }, []);\\n \\n const tools = [\\n { tool: \'Checkpointz\', version: \'v0.27.0\', repo: \'checkpointz\' },\\n { tool: \'Contributoor\', version: \'v0.0.63\', repo: \'contributoor\' },\\n { tool: \'Dora\', version: \'v1.15.2\', repo: \'dora\' },\\n { tool: \'Xatu\', version: \'v1.0.49\', repo: \'xatu\' },\\n { tool: \'Tracoor\', version: \'v0.0.29\', repo: \'tracoor\' },\\n { tool: \'Ethereum Metrics Exporter\', version: \'v0.27.1\', repo: \'ethereum-metrics-exporter\' },\\n { tool: \'Go Beacon Package\', version: \'v0.51.0\', repo: \'beacon\' }\\n ];\\n\\n return (\\n <div style={{ \\n width: \'100%\',\\n marginBottom: \'24px\',\\n position: \'relative\'\\n }}>\\n <div style={{ \\n width: \'100%\',\\n overflowX: isMobile ? \'auto\' : \'visible\',\\n WebkitOverflowScrolling: \'touch\'\\n }}>\\n <div style={{ \\n minWidth: \'100%\',\\n display: \'flex\',\\n flexDirection: \'column\',\\n borderRadius: \'6px\',\\n overflow: \'hidden\',\\n border: \'1px solid var(--ifm-color-emphasis-200)\',\\n }}>\\n {/* Table Header */}\\n <div style={{\\n display: \'flex\', \\n backgroundColor: \'var(--ifm-color-primary)\',\\n color: \'white\',\\n padding: \'10px 16px\',\\n fontWeight: \'bold\',\\n fontSize: \'0.95rem\'\\n }}>\\n <div style={{flex: \'0 0 40%\'}}>Tool</div>\\n <div style={{flex: \'0 0 30%\'}}>Minimum Version</div>\\n <div style={{flex: \'0 0 30%\', textAlign: \'center\'}}>Link</div>\\n </div>\\n \\n {/* Table Rows */}\\n {tools.map((tool, index) => (\\n <div \\n key={index}\\n style={{\\n display: \'flex\',\\n backgroundColor: index % 2 === 0 ? \'var(--ifm-table-stripe-background)\' : \'var(--ifm-background-surface-color)\',\\n padding: \'10px 16px\',\\n borderBottom: index < tools.length - 1 ? \'1px solid var(--ifm-color-emphasis-200)\' : \'none\',\\n fontSize: \'0.9rem\'\\n }}\\n >\\n <div style={{flex: \'0 0 40%\'}}>{tool.tool}</div>\\n <div style={{flex: \'0 0 30%\'}}>{tool.version}</div>\\n <div style={{flex: \'0 0 30%\', textAlign: \'center\'}}>\\n <a \\n href={`https://github.com/ethpandaops/${tool.repo}`}\\n style={{\\n color: \'var(--ifm-color-primary)\',\\n textDecoration: \'none\',\\n fontWeight: \'medium\'\\n }}\\n target=\\"_blank\\"\\n rel=\\"noopener noreferrer\\"\\n >\\n GitHub\\n </a>\\n </div>\\n </div>\\n ))}\\n </div>\\n </div>\\n </div>\\n );\\n};\\n\\n<ToolingTable />
1\\n\\n## Monitoring Resources\\n\\nEthereum is a complex system, and it can be overwhelming to keep track of all the moving parts. We\'ve got you covered with a list of key events to watch out for:\\n\\nexport const MonitoringResources = () => {\\n const [isMobile, setIsMobile] = useState(false);\\n \\n useEffect(() => {\\n const checkIfMobile = () => {\\n setIsMobile(window.innerWidth < 768);\\n };\\n \\n checkIfMobile();\\n window.addEventListener(\'resize\', checkIfMobile);\\n \\n return () => {\\n window.removeEventListener(\'resize\', checkIfMobile);\\n };\\n }, []);\\n \\n const events = [\\n { \\n event: \'Electra activation\', \\n description: \'The Electra activation is the first step in the upgrade process, and is the point at which the network will begin to produce Electra blocks.\',\\n whereToWatch: <a href=\\"https://www.youtube.com/watch?v=NbxaCZYtW-I\\">Pectra Upgrade Livestream</a>\\n },\\n { \\n event: \'Electra blocks\',\\n description: \'Block proposals are the first indicator that validators on the network successfully upgraded, and a high proposal rate is a good first sign that the network is healthy.\',\\n whereToWatch: <>\\n <a href=\\"https://lab.ethpandaops.io/beacon/slot/live\\">The Lab</a>, <a href=\\"https://beaconcha.in/\\">Beaconcha.in</a>, <a href=\\"https://ethseer.io/?network=mainnet\\">EthSeer</a>\\n </>\\n },\\n { \\n event: \'Electra attestations\',\\n description: \'This time we\\\\\'re watching for attestations to be produced. This means that other validators on the network were able to import, verify, and attest to the proposed Electra blocks. If we see a high attestation rate, it\\\\\'s a very good sign that the network is healthy.\',\\n whereToWatch: <>\\n <a href=\\"https://lab.ethpandaops.io/beacon/slot/live\\">The Lab</a>, <a href=\\"https://beaconcha.in/\\">Beaconcha.in</a>, <a href=\\"https://ethseer.xyz/\\">EthSeer</a>\\n </>\\n },\\n { \\n event: \'Blocks and blobs arriving on time\',\\n description: <>Blocks, and their associated blobs, need to be seen by 66% of the validators in the network otherwise they run the risk of being orphaned. <a href=\\"/posts/contribute-to-xatu-data/\\">Thanks to our wonderful contributoors</a>, we have a real-time view of the network and can see if blocks and blobs are arriving on time. This is especially important with the inclusion of <a href=\\"https://eips.ethereum.org/EIPS/eip-7691\\">EIP-7691</a>, which increases blob throughput.</>,\\n whereToWatch: <a href=\\"https://lab.ethpandaops.io/beacon/slot/live\\">The Lab</a>\\n },\\n { \\n event: \'ethPandaOps blocks built locally\',\\n description: \'Our team runs multiple nodes on the network, and we constantly build \\"throw away\\" blocks to test all clients\\\\\' ability to build blocks for the current slot. This gives us immediate feedback on any issues that may arise with clients, reducing the response time for client developers!\',\\n whereToWatch: <a href=\\"https://lab.ethpandaops.io/beacon/locally-built-blocks\\">The Lab</a>\\n },\\n { \\n event: \'Terence puts the champagne away \ud83c\udf7e\',\\n description: \'Terence has a reputation for declaring victory moments before chaos, so hopefully he has the day off Twitter.\',\\n whereToWatch: <a href=\\"https://x.com/terencechain\\">Terence\'s Twitter</a>\\n },\\n { \\n event: \'Justification and Finalization\',\\n description: \'It\\\\\'s in the name! Once the network has finalized, Pectra has been locked in.\',\\n whereToWatch: <a href=\\"https://beaconcha.in/\\">Beaconcha.in</a>\\n },\\n { \\n event: \'Feature tests\',\\n description: \'Some of the Pectra specific features, such as consolidations and deposits will take a longer period of time to test. The testing and client teams will be looking through everything that happens with a fine comb and announce the success of the fork once they are done!\',\\n whereToWatch: <a href=\\"https://x.com/ethpandaops/\\">ethPandaOps\'s Twitter</a>\\n }\\n ];\\n\\n return (\\n <div style={{ \\n width: \'100%\',\\n marginBottom: \'24px\',\\n position: \'relative\'\\n }}>\\n <div style={{ \\n width: \'100%\',\\n overflowX: isMobile ? \'auto\' : \'visible\',\\n WebkitOverflowScrolling: \'touch\'\\n }}>\\n <div style={{ \\n minWidth: \'100%\',\\n display: \'flex\',\\n flexDirection: \'column\',\\n borderRadius: \'6px\',\\n overflow: \'hidden\',\\n border: \'1px solid var(--ifm-color-emphasis-200)\'\\n }}>\\n {/* Table Header \u2014 hidden on mobile, where rows stack into cards */}\\n <div style={{\\n display: isMobile ? \'none\' : \'flex\', \\n backgroundColor: \'var(--ifm-color-primary)\',\\n color: \'white\',\\n padding: \'10px 16px\',\\n fontWeight: \'bold\',\\n fontSize: \'0.95rem\'\\n }}>\\n <div style={{flex: \'0 0 25%\', paddingRight: \'20px\'}}>Event</div>\\n <div style={{flex: \'0 0 75%\'}}>Description</div>\\n </div>\\n \\n {/* Table Rows */}\\n {events.map((event, index) => (\\n <div \\n key={index}\\n style={{\\n display: \'flex\',\\n flexDirection: isMobile ? \'column\' : \'row\',\\n backgroundColor: index % 2 === 0 ? \'var(--ifm-table-stripe-background)\' : \'var(--ifm-background-surface-color)\',\\n padding: \'10px 16px\',\\n borderBottom: index < events.length - 1 ? \'1px solid var(--ifm-color-emphasis-200)\' : \'none\',\\n fontSize: \'0.9rem\'\\n }}\\n >\\n <div style={{flex: isMobile ? \'1 1 auto\' : \'0 0 25%\', paddingRight: isMobile ? \'0\' : \'20px\', marginBottom: isMobile ? \'4px\' : \'0\'}}
1>\\n <strong>{event.event}</strong>\\n </div>\\n <div style={{flex: isMobile ? \'1 1 auto\' : \'0 0 75%\'}}>\\n <div>{event.description}</div>\\n <div style={{\\n marginTop: \'8px\',\\n display: \'flex\',\\n alignItems: \'center\',\\n fontSize: \'0.85rem\',\\n color: \'var(--ifm-color-emphasis-700)\'\\n }}>\\n <span style={{\\n backgroundColor: \'var(--ifm-color-emphasis-200)\',\\n borderRadius: \'12px\',\\n padding: \'3px 10px\',\\n marginRight: \'6px\',\\n display: \'inline-flex\',\\n alignItems: \'center\'\\n }}>\\n Where to Watch: {event.whereToWatch}\\n </span>\\n </div>\\n </div>\\n </div>\\n ))}\\n </div>\\n </div>\\n </div>\\n );\\n};\\n\\n<MonitoringResources />\\n\\n## Thank you\\n\\nThis upgrade is a monumental achievement for the Ethereum ecosystem. After a full year of hard work from Core Devs, we\'re beyond excited to see it go live. A massive thank you to everyone who\'s been part of this journey with us.\\n\\nWe want to give particular thanks to the contributoors [sharing their node data with the ethPandaOps team](/posts/contribute-to-xatu-data/). This data gives us a view into the network we\'ve never had before during critical forks. It helps us jump on problems quickly and to sleep better at night, while also allowing the wider community to be more informed when making decisions about future upgrades.\\n\\nSee you on the other side!\\n\\n<center>\\nWith love,\\n\\n[parithosh, skylenet, samcm, savid, barnabasbusa, pk910 & matty](/team)\\n\\n:heart:\\n</center>"},{"id":"hoodi-attestation-packing","metadata":{"permalink":"/posts/hoodi-attestation-packing","source":"@site/blog/2025-04-02-hoodi-attestation-packing/index.md","title":"Electra on Hoodi: Attestation Packing","description":"Analysis of the Hoodi Electra fork and its impact on attestation packing.","date":"2025-04-02T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"consensus-layer","permalink":"/posts/tags/consensus-layer"},{"inline":true,"label":"hoodi","permalink":"/posts/tags/hoodi"},{"inline":true,"label":"electra","permalink":"/posts/tags/electra"}],"readingTime":5.59,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"hoodi-attestation-packing","title":"Electra on Hoodi: Attestation Packing","authors":["samcm"],"description":"Analysis of the Hoodi Electra fork and its impact on attestation packing.","tags":["xatu","consensus-layer","hoodi","electra"],"image":"/img/blog/hoodi-attestations.jpg","relatedLinks":[{"name":"EIP-7549","url":"https://eips.ethereum.org/EIPS/eip-7549"},{"name":"Prysm Analysis","url":"https://hackmd.io/@ttsao/attestation-packing-pectra"},{"name":"Jupyter Notebook","url":"https://github.com/ethpandaops/xatu-data/blob/e9b7ef41ada2cbbb44d790994614a9aa302512fc/analysis/attestations/packing.ipynb"}]},"unlisted":false,"prevItem":{"title":"Pectra Mainnet Checklist","permalink":"/posts/pectra-mainnet-checklist"},"nextItem":{"title":"EthPandaOps Pivots to GitChain","permalink":"/posts/april-fools"}},"content":"import Alert from \'@site/src/components/Alert\';\\nimport ClickableImage from \'@site/src/components/ClickableImage\';\\n\\nLast week, the newly established Hoodi network successfully activated its Electra fork, significantly changing how attestations are packed into beacon blocks, as detailed in [EIP-7549](https://eips.ethereum.org/EIPS/eip-7549). This post recaps how attestation aggregation worked previously and how the recent fork changes things, before diving into some analysis.\\n\\n<Alert>\\nCheck out Prysm\'s Terence\'s post for another detailed analysis of [Attestation performance in Hoodi](https://hackmd.io/@ttsao/attestation-packing-pectra).\\n</Alert>
1\\n\\n## Background\\n\\n### Before Electra\\n\\nPreviously, attestations included the `committee_index` within the signed data.\\n\\n- Attestations could only be aggregated within their specific committee, limiting overall aggregation efficiency.\\n- Identical votes from different committees produced distinct signatures.\\n\\n### After Electra\\n\\nWith EIP-7549, the attestation structure was changed by moving the `committee_index` field **outside** the signed data.\\n\\n- Identical votes across different committees can now be aggregated, drastically cutting down on unique attestations.\\n- Significant efficiency improvements in signature verification e.g., in a network with 262,144 validators, attestations required for a two-thirds majority fell dramatically from approximately 1,366 to just around 22.\\n\\nWhile these changes offer great potential, there is also a risk that the changes will negatively impact performance. With these changes outlined, let\u2019s explore how Electra unfolded on Hoodi.\\n\\n## Analysis\\n\\n<Alert>\\nThe raw Jupyter notebook used to generate the analysis below is available [here](https://github.com/ethpandaops/xatu-data/blob/e9b7ef41ada2cbbb44d790994614a9aa302512fc/analysis/attestations/packing.ipynb).\\n</Alert>\\n\\n### Methodology\\n\\nThe following parameters were used for the analysis:\\n\\n- 2x 6 hour time periods:\\n - Pre-Electra: `2025-03-25T12:10:00Z` to `2025-03-25T18:10:00Z`\\n - Post-Electra: `2025-03-27T12:10:00Z` to `2025-03-27T18:10:00Z`\\n- Data was collected from the Hoodi network using [Xatu](/data/xatu/) (keep an eye out for public Hoodi data soon!).\\n- The analysis looks at canonical beacon blocks and inspected the attestations that were included in them.\\n - This analysis examines attestation packing efficiency (and by proxy aggregation performance), rather than evaluating the correctness of attestation votes.\\n- We naively assume that client teams are running the client they develop.\\n - Nimbus keys, managed by the ethPandaOps team during the analysis period, exclusively utilized the Nimbus beacon node client.\\n- The ethPandaOps validator keys use multiple beacon node clients.\\n\\n### Results\\n\\n#### Unique Validators Per Block\\n\\nOur first metric examines the **unique count of validators** included per block aggregated across all blocks during the two time periods.\\n\\nWith the Electra fork, we\'re expecting to see a much higher number of unique validators per block as older attestations can now be included. Note that this metric includes stale attestations by design.\\n\\n<ClickableImage alt=\\"Unique Validators Per Block\\" src=\\"/img/blog/all-unique-validators-per-block.jpg\\" />\\n\\nOur analysis confirms a notable increase in unique validator indices per block, with some blocks including as many as 200,000 unique validators.\\n\\n#### Fresh Attestations\\n\\nThe second metric assesses the number of fresh attestations per block. A fresh attestation is defined as a validator vote that has not been seen in any previous block. As attestations land in beacon blocks _aggregated_, it\'s inevitable that some stale attestations will be included as overlaps can occur. However, we want to include as many fresh attestations as possible.\\n\\n<ClickableImage alt=\\"Fresh Attestations\\" src=\\"/img/blog/all-first-seen-attestations.jpg\\" />\\n\\nThe analysis confirms a clear and substantial increase in capacity due to EIP-7549. With 1.08 million validators on Hoodi, approximately 33.7k attestations can occur per slot. The 80th percentile of Electra blocks aligns closely with this number, while a notable jump to approximately 60k attestations at the 90th percentile occurs--presumably after a missed slot.\\n\\nSurprisingly, a few blocks included as many as 120k fresh attestations. While this increased capacity is great, it\'s also a sign that the network may be experiencing issues. Statistically, the chance of observing four consecutive missed slots at 90% participation is around 1 in 10,000. Given our analysis covers only 3,276 blocks, it is unlikely we observed such an event.\\n\\n<ClickableImage alt=\\"Missed Slots\\" src=\\"/img/blog/missed-slots.jpg\\" />
1\\n\\nConfirming that we didn\'t see any blocks with 4 missed slots in a row, we can now ask the question: **why are block proposers ever given the chance to include fresh attestions older than 1, 2 or 3 missed slots?**\\n\\n#### Inclusion Distance\\n\\nInclusion distance measures the number of slots between the attestation\u2019s creation and its inclusion in a block. In a perfect world, the inclusion distance should be one slot.\\n\\n<ClickableImage alt=\\"Inclusion Distance\\" src=\\"/img/blog/all-inclusion-delay.jpg\\" />\\n\\nThe analysis reveals Nimbus client blocks experienced notably increased inclusion distances post-Electra, suggesting potential improvements in packing strategies.\\n\\n#### Optimal Inclusion Distance\\n\\nThe Optimal Inclusion Distance metric counts the number of attestations in the block with an inclusion distance of just one slot, highlighting the ability of block proposers to capture the most recent attestations.\\n\\n> Note: this is slightly different to [ethdo\'s](https://github.com/wealdtech/ethdo/blob/25a5bd917f55c1802a5bd924cba98bc56bf82c9b/cmd/epoch/summary/process.go#L381) _head timely_ metric which additionally looks at the correctness of the vote.\\n\\n<ClickableImage alt=\\"Optimal Inclusion Distance\\" src=\\"/img/blog/head-timely-validators.jpg\\" />\\n\\nIt appears that Nimbus experienced a larger regression than other clients - warranting further investigation. Grandine is also seemingly performing worse that others, although the change before/after the fork is not as large.\\n\\n<ClickableImage alt=\\"Head Timely Nimbus\\" src=\\"/img/blog/head-timely-nimbus.jpg\\" />\\n\\nDiving in to just Nimbus produced blocks, we can see that around 40% of them failed to include any of the most recent attestations. This outlier hints that there may be room for improvement with the Nimbus attestation packing strategy.\\n\\n<ClickableImage alt=\\"Head Timely Ex. Nimbus\\" src=\\"/img/blog/head-timely-no-nimbus.jpg\\" />\\n\\nLooking at the same metric at the network level while excluding Nimbus, we see that peformance is still around 6% worse.\\n\\nThis is somewhat concerning. It\'s hard to attribute this to a drop in network participation, which remained around 90% [before](https://dora.hoodi.ethpandaops.io/epoch/1800) and [after](https://dora.hoodi.ethpandaops.io/epoch/2250) the fork. This may hint at underlying inefficiencies with attestation aggregation across the network before they get to the block proposer, but is out of scope for this analysis. It\'s also very likely that clients are still fine tuning their attestation packing strategies, and we expect to see this metric improve over time, particularly before Electra hits mainnet.\\n\\n### Conclusion\\n\\nThe Electra fork has enhanced the capacity of attestations being included into beacon blocks. However, these advancements highlight the increased importance of efficient aggregation and proactive packing strategies.\\n\\nWhile we observed some mixed results, the changes introduced in [EIP-7549](https://eips.ethereum.org/EIPS/eip-7549) are promising, and we expect the changes to be a net positive for the beacon chain in the long term, especially in times of turbulence."},{"id":"april-fools","metadata":{"permalink":"/posts/april-fools","source":"@site/blog/2025-04-01-april-fools/index.md","title":"EthPandaOps Pivots to GitChain","description":"After careful consideration and extensive technical analysis, EthPandaOps is proud to announce our full support for GitChain.","date":"2025-04-01T00:00:00.000Z","tags":[{"inline":true,"label":"ethereum","permalink":"/posts/tags/ethereum"},{"inline":true,"label":"upgrade","permalink":"/posts/tags/upgrade"},{"inline":true,"label":"gitchain","permalink":"/posts/tags/gitchain"}],"readingTime":2.03,"hasTruncateMarker":false,"authors":[{"name":"ethPandaOps Team","title":"ethPandaOps Team","url":"https://github.com/ethpandaops","imageURL":"https://github.com/ethpandaops.png","key":"team","page":null}],"frontMatter":{"slug":"april-fools","title":"EthPandaOps Pivots to GitChain","authors":["team"],"description":"After careful consideration and extensive technical analysis, EthPandaOps is proud to announce our full support for GitChain.","tags":["ethereum","upgrade","gitchain"],"image":"/img/blog/april-fools.jpg","relatedLinks":[{"name":"GitChain EIP","url":"https://github.com/ethereum/EIPs/pull/9579"},{"name":"Ethereum Magicians Forum","url":"https://ethereum-magicians.org/t/eip-gitchain/23336"}]},"unlisted":false,"prevItem":{"title":"Electra on Hoodi: Attestation Packing","permalink":"/posts/hoodi-attestation-packing"},"nextItem":{"title":"Contributoor: A Lightweight Beacon Node Companion","permalink":"/posts/contributoor-beacon-node-companion"}},"content":"After careful consideration and extensive technical analysis, EthPandaOps is proud to announce our full support for [GitChain](https://github.com/ethereum/EIPs/pull/9579), a groundbreaking proposal that will revolutionize Ethereum\'s execution layer by leveraging Git as the primary execution engine.\\n\\n<img src=\\"/img/blog/pivot.jpg\\" />
1\\n\\n## Why GitChain?\\n\\nThe advantages of GitChain are compelling and numerous:\\n\\n1. **Immediate Post-Quantum Security**: By eliminating traditional signatures in favor of GitHub account authentication, GitChain provides inherent resistance to quantum computing attacks.\\n\\n2. **MEV Resistance**: The sequential processing of transactions based on GitHub issue numbers introduces a deterministic ordering mechanism that effectively eliminates MEV extraction opportunities.\\n\\n3. **Developer Familiarity**: With over 94% of Ethereum developers already using Git daily, the learning curve for GitChain will be minimal compared to other proposed solutions.\\n\\n4. **Simplified Infrastructure**: By utilizing GitHub\'s robust infrastructure, we can significantly reduce the complexity of running nodes while leveraging GitHub\'s proven 99.99% uptime.\\n\\n## Implementation Timeline\\n\\nEthPandaOps is committed to supporting this transition with the following roadmap:\\n\\n- Q2 2025: Initial testnet deployment\\n- Q3 2025: Security audits and GitHub API integration testing\\n- Q4 2025: Mainnet shadow fork\\n- Q1 2026: Full mainnet deployment\\n\\n## Technical Details\\n\\nThe GitChain protocol elegantly solves several long-standing challenges in blockchain technology:\\n\\n- Transaction ordering is determined by GitHub issue numbers\\n- Block production is managed through pull requests\\n- State transitions are handled via Git commits\\n- Consensus is achieved through GitHub\'s merge button\\n- Smart contracts are deployed as GitHub Actions workflows\\n\\n## Infrastructure Impact\\n\\nOur analysis shows that GitChain will reduce node operation complexity by approximately 87%. Node operators will only need to maintain a GitHub API key and a stable internet connection, significantly lowering the barrier to entry for network participation.\\n\\n## Community Support\\n\\nWe are pleased to note that several key ecosystem participants have already expressed their enthusiasm for this proposal, with one reviewer noting that this solution \\"solves MEV\\" and another highlighting its \\"fantastic\\" nature as an alternative to other proposed execution layers.\\n\\n## Next Steps\\n\\nEthPandaOps will be organizing a series of workshops and technical sessions to help the community understand and prepare for this transformative upgrade. We encourage all stakeholders to review the EIP and participate in the discussion at the [Ethereum Magicians forum](https://ethereum-magicians.org/t/eip-gitchain/23336).\\n\\nWe believe that GitChain represents the natural evolution of blockchain technology, combining the best of Web2 infrastructure with Web3 principles. We look forward to working with the community to make this vision a reality."},{"id":"contributoor-beacon-node-companion","metadata":{"permalink":"/posts/contributoor-beacon-node-companion","source":"@site/blog/2025-01-27-contributoor-beacon-node-companion/index.md","title":"Contributoor: A Lightweight Beacon Node Companion","description":"We\'re excited to announce Contributoor; a new monitoring and data-gathering tool that helps improve Ethereum\'s network visibility while running seamlessly alongside your beacon node.","date":"2025-01-27T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"consensus layer","permalink":"/posts/tags/consensus-layer"},{"inline":true,"label":"contributoor","permalink":"/posts/tags/contributoor"},{"inline":true,"label":"data","permalink":"/posts/tags/data"},{"inline":true,"label":"L1","permalink":"/posts/tags/l-1"}],"readingTime":3.03,"hasTruncateMarker":false,"authors":[{"name":"matty","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/mattevans","github":"https://github.com/mattevans","twitter":"https://x.com/mattevansnz","imageURL":"/img/team/matty.jpeg","key":"matty","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null}],"frontMatter":{"slug":"contributoor-beacon-node-companion","title":"Contributoor: A Lightweight Beacon Node Companion","description":"We\'re excited to announce Contributoor; a new monitoring and data-gathering tool that helps improve Ethereum\'s network visibility while running seamlessly alongside your beacon node.","authors":["matty","samcm","savid"],"tags":["xatu","consensus layer","contributoor","data","
1L1"],"image":"/img/blog/contributoor-beacon-node-companion.jpg","githubRepos":[{"name":"contributoor","url":"https://github.com/ethpandaops/contributoor"},{"name":"contributoor-installer","url":"https://github.com/ethpandaops/contributoor-installer"}],"relatedLinks":[{"name":"Xatu Lab Dashboard","url":"https://lab.ethpandaops.io/xatu"},{"name":"Xatu Telegram Group","url":"https://t.me/+JanoQFu_nO8yNzQ1"}]},"unlisted":false,"prevItem":{"title":"EthPandaOps Pivots to GitChain","permalink":"/posts/april-fools"},"nextItem":{"title":"Understanding the Ethereum network limits using devnets","permalink":"/posts/network-limit-devnets"}},"content":"We\'re excited to announce **[Contributoor](https://github.com/ethpandaops/contributoor)**, a new monitoring and data-gathering tool that helps improve Ethereum\'s network visibility while running seamlessly alongside your beacon node. Think of it as a streamlined version of [Xatu Sentry](https://ethpandaops.io/posts/contribute-to-xatu-data/) with a smaller footprint, fewer dependencies, and a much simpler setup process. For more info on Xatu Sentry, check out our [post on contributing to the Xatu dataset](https://ethpandaops.io/posts/contribute-to-xatu-data/).\\n\\n### Why Contributoor?\\n\\nWe\'re building a network of home stakers to provide comprehensive and decentralized timing data for Ethereum. We started with a small group, and contributoor is a step towards including the broader community as our primary data source.\\n\\n- **Lightweight**: This tool is slimmed down and focuses on the essential features you need.\\n- **Easy Installation**: We built an installer\u2014[Contributoor Installer](https://github.com/ethpandaops/contributoor-installer)\u2014so anyone can get started in minutes.\\n- **Reduced Complexity**: Contributoor is set up to get you running faster, with minimal overhead compared to a typical Xatu Sentry installation.\\n\\n### How It Works\\n\\nContributoor runs as a sidecar process to your beacon node, collecting important metrics and events without impacting your node\'s performance. It:\\n\\n- Connects to your beacon node\'s API\\n- Monitors key events and metrics\\n- Securely forwards data to our collection endpoints\\n\\n### What Does It Do?\\n\\nContributoor captures the same [events Xatu Sentry does](https://ethpandaops.io/posts/contribute-to-xatu-data/#events-collected), including:\\n\\n- `beacon_api_eth_v1_events_block`\\n- `beacon_api_eth_v1_events_blob_sidecar`\\n- `beacon_api_eth_v1_events_chain_reorg`\\n- `beacon_api_eth_v1_events_finalized_checkpoint`\\n- `beacon_api_eth_v1_events_head_v2`\\n\\nThis data helps researchers and developers:\\n\\n- Optimize network performance\\n- Identify potential issues early\\n- Improve client implementations\\n- Understand network behavior during upgrades\\n\\n### Getting Started\\n**Installation**:\\n```bash\\n# Download the script.\\ncurl -O https://raw.githubusercontent.com/ethpandaops/contributoor-installer/refs/heads/master/install.sh\\n\\n# Inspect the script contents.\\nless install.sh\\n\\n# Make it executable and run if you\'re satisfied with the contents.\\nchmod +x install.sh && ./install.sh\\n```\\n\\nAlternatively, if you trust the source, you can run this one-liner:\\n```bash\\ncurl -O https://raw.githubusercontent.com/ethpandaops/contributoor-installer/refs/heads/master/install.sh && chmod +x install.sh && ./install.sh\\n```\\n\\n> **Using eth-docker?** Contributoor can be run as part of your eth-docker setup - [check it out here](https://github.com/ethpandaops/contributoor-installer?tab=readme-ov-file#getting-started).\\n\\n**Managing Contributoor**:\\n\\nAfter installation, you can manage Contributoor using a single command-line interface:\\n\\n```bash\\ncontributoor start # Start the service\\ncontributoor stop # Stop the service\\ncontributoor status # Check service status\\ncontributoor restart # Restart the service\\ncontributoor config # View/edit configuration\\ncontributoor update # Update to latest version\\ncontributoor logs # Show logs\\n```\\n\\nFor more details or troubleshooting tips, check out the main [Contributoor GitHub repository](https://github.com/ethpandaops/contributoor). \\n\\n### Staying Up to Date\\n\\nIt\'s a good idea to periodically check contributoor to ensure you\'re on the latest version!\\n\\nKeeping up to date is simple - just run:\\n\\n```bash\\ncontributoor update\\n```\\n\\n\\n\\n### Ready to Contribute?\\n\\nBy offering Contributoor as a streamlined data collection tool, we hope to empower the Ethereum community to gather valuable insights and drive meaningful improvements to the network. \\n\\nWe\'re gradually scaling up our data collection network, starting with home stakers. If you run your own beacon node and want to help improve Ethereum\'s network visibility:\\n\\n- **Who Can Join*
1*: For now, individual home stakers running their own beacon nodes\\n- **Getting Started**: We need some basic details to get you started - [Apply here](/contribute-data/)\\n- **What\'s Next**: We\'re expanding access as we scale, with home stakers being our primary focus\\n\\n### Get in Touch\\n\\n- Join our [Telegram](https://t.me/+JanoQFu_nO8yNzQ1) group\\n- Find us on [Twitter](https://x.com/ethpandaops)\\n\\n> \ud83e\udd2b Psst... we\'ve got something cooking in the [lab \ud83e\uddea](https://lab.ethpandaops.io/xatu) that lets you peek at some of the contributed data. More on that soon!"},{"id":"network-limit-devnets","metadata":{"permalink":"/posts/network-limit-devnets","source":"@site/blog/2024-11-27-network-limit-devnets/index.md","title":"Understanding the Ethereum network limits using devnets","description":"The Dencun upgrade on Ethereum introduced the concept of blob. These blobs represent data that every node is required to store for a defined period of time. With the discussion about a blob increase in Pectra, lets review results from a network run with various edge cases.","date":"2024-11-27T00:00:00.000Z","tags":[{"inline":true,"label":"ansible","permalink":"/posts/tags/ansible"},{"inline":true,"label":"ethereum","permalink":"/posts/tags/ethereum"},{"inline":true,"label":"devnets","permalink":"/posts/tags/devnets"},{"inline":true,"label":"L1","permalink":"/posts/tags/l-1"},{"inline":true,"label":"testnet","permalink":"/posts/tags/testnet"}],"readingTime":4.34,"hasTruncateMarker":false,"authors":[{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null},{"name":"barnabasbusa","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/barnabasbusa","github":"https://github.com/barnabasbusa","twitter":"https://x.com/BarnabasBusa","imageURL":"/img/team/barnabasbusa.jpeg","key":"barnabasbusa","page":null},{"name":"skylenet","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/skylenet","github":"https://github.com/skylenet","twitter":"https://x.com/skylenet","imageURL":"/img/team/skylenet.jpeg","key":"skylenet","page":null},{"name":"pk910","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/pk910","github":"https://github.com/pk910","twitter":"https://x.com/_pk910_","imageURL":"/img/team/pk910.jpeg","key":"pk910","page":null}],"frontMatter":{"slug":"network-limit-devnets","title":"Understanding the Ethereum network limits using devnets","authors":["parithosh","barnabasbusa","skylenet","pk910"],"description":"The Dencun upgrade on Ethereum introduced the concept of blob. These blobs represent data that every node is required to store for a defined period of time. With the discussion about a blob increase in Pectra, lets review results from a network run with various edge cases.","tags":["ansible","ethereum","devnets","L1","testnet"],"image":"/img/blog/network-limit-devnets.jpg","relatedLinks":[{"name":"Pectra EIP-7600","url":"https://eips.ethereum.org/EIPS/eip-7600"},{"name":"ProbeLab Analysis","url":"https://probelab.io/ethereum/discv5/2024-46/#cloud-hosting-rate"}]},"unlisted":false,"prevItem":{"title":"Contributoor: A Lightweight Beacon Node Companion","permalink":"/posts/contributoor-beacon-node-companion"},"nextItem":{"title":"Xatu Execution Layer data now available","permalink":"/posts/xatu-execution-layer"}},"content":"## Intent\\nThe [Dencun](https://ethereum.org/en/roadmap/dencun/) upgrade on Ethereum introduced the concept of blobs into Ethereum. These blobs represent data that every node is required to store for a defined period of time. With the discussion about a blob increase in [Pectra](https://eips.ethereum.org/EIPS/eip-7600), we wanted to run a network in various edge cases to find out potential problems we may face. Two scenarios that were deemed important to test is long range syncing after a non-finality period as well as a network with resource constrained nodes.\\n\\n## General setup\\n- 1M validators, 516 hosts (geodistributed across 10 regions)\\n- 8
1c/32GB RAM to ensure enough resources for the client\\n- Mainnet client distribution\\n- Healthy blob and transaction gossip (average blob/slot of 3)\\n\\n## Experiments\\n1. 5% Gigabit nodes, 95% nodes with 20mbps/10mbps download/upload connections.\\n2. 80% Gigabit nodes, 20% nodes with 100mbps/20mbps download/upload connections. Non-finality for ~1d and then nodes are started back up to sync.\\n3. A flat network with every node containing 100mbps/50mbps download/upload connections. Non-finality for ~12h and then nodes are started back up to sync.\\n4. A flat network with every node containing 50mbps/25mbps download/upload connections.\\n5. A flat network with every node containing 30mbps/15mbps download/upload connections. Some nodes were turned off for 2h to test syncing times.\\n6. A flat network with every node containing 20mbps/10mbps download/upload connections.\\n\\n## Aggregated learning\\n- Our mental model of the Ethereum p2p layer probably needs some reworking, our mental model of a flat network with equal low bandwidth nodes probably isn\'t true for our current DA setup. The analysis by [ProbeLabs](https://probelab.io/ethereum/discv5/2024-46/#cloud-hosting-rate) also indicate a hybrid network of faster and slower nodes. A flat network style test is probably still helpful to understand how to design upgrades for the worst case scenario.\\n- Long range sync times after a period of non-finality seem to be reasonable and the found problems potentially lie in peering bugs or range request optimisations.\\n- The bottleneck for the client at 20mbps/10mbps seems to be fetching data to validate DA within the slot time limits (perhaps due to peer rate limits or other limitations). At gigabit links however, the DA data fetching bottlenecks seems to be solved. But the node still takes ~20mins to sync to head (the theoretical line speed to download the data would take 20s). The bottleneck for long range sync doesn\u2019t seem to be raw bandwidth related(Most nodes were peaking at 100mbps traffic, so they had a 10x bandwidth headroom), but instead seems to be batch verification/peering/rate limiting/peer download limit related and we might need to narrow this down in future tests.\\n- Even in a network with mostly gigabit nodes, no node was using more than ~160mbps of traffic. This meant the gigabit nodes still had a ~5x of bandwidth as headroom to serve data.\\n- In a network with 100mbps/50mbps rate limits applied, The nodes were all hitting the bandwidth usage of 50mbps (both up and down). This is despite the fact that the download bandwidth is capped at 100mbps and indicates a potential bottleneck in fetching data from various peers instead of a single peer.\\n- The network starts to only degrade meaningfully once we go below 30/15mbps download/upload speeds. The total vote might remain high, but we do observe a lower head vote percentage. The degradation is quite noticeable at a flat network of 20mbps/10mbps download/upload speeds, while the network is still able to finalize.\\n- There seem to be optimisations required in batch fetching, rate limits as well as potential bugs in peering that are triggered in these stressful scenarios.\\n- The devnet presents a \\"clean\\" EL state with no real EL load or limitation. A potential future experiment can try a mainnet shadowfork to find EL related bottlenecks. Estimations on mainnet indicate that average this would add about 20 seconds per batch, or an acceptable overhead of 300ms per block.\\n\\n## Conclusion\\n- The network indicates that clients are not purely bottlenecked by bandwidth, instead they can still apply some optimisations to help improve syncing and peering performance. These optimisations might come with other tradeoffs and we need to study potential paths forward.\\n- These optimisations could already help mainnet today and would be needed irrespective of our decision to increase the blob count from the current 3/6 limits.\\n- Long range syncing after a period of non-finality is able to be completed in a reasonable amount of time and would benefit from some optimisations/peering bug fixes.\\n- The analysis done does not assume any blob censorship or timing games.In events of high volatility, we might see more builder censorship. If we were to increase the max, we would likely see patterns similar to [0,0,max,max,0,0,0,max,max,max], etc., which is different from the common steady rate of target blobs per slot.\\n- We need more studies on the benefit the `get-blobs` optimisation has already provided and potentially could provide\\n- We need to have bandwidth contraints on more feature devnets as well, probably as a default on our stack based on mainnet measurements"},{"id":"xatu-execution-layer","metadata":{"permalink":"/posts/xatu-execution-layer","source":"@site/blog/2024-10-16-xatu-execution-layer/index.md","title":"Xatu Execution Layer data now available","description":"We\'re excited to announce that E
1xecution layer data, extracted via cryo, is now available in the Xatu dataset.","date":"2024-10-16T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"execution layer","permalink":"/posts/tags/execution-layer"},{"inline":true,"label":"cryo","permalink":"/posts/tags/cryo"}],"readingTime":4.42,"hasTruncateMarker":false,"authors":[{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"xatu-execution-layer","title":"Xatu Execution Layer data now available","authors":["savid","samcm"],"tags":["xatu","execution layer","cryo"],"image":"/img/blog/xatu-execution-layer.jpg","githubRepos":[{"name":"xatu-data","url":"https://github.com/ethpandaops/xatu-data"},{"name":"paradigmxyz/cryo","url":"https://github.com/paradigmxyz/cryo"}],"relatedLinks":[{"name":"Apache Parquet","url":"https://parquet.apache.org"},{"name":"Xatu Telegram Group","url":"https://t.me/+JanoQFu_nO8yNzQ1"}]},"unlisted":false,"prevItem":{"title":"Understanding the Ethereum network limits using devnets","permalink":"/posts/network-limit-devnets"},"nextItem":{"title":"Contribute to Xatu: Join the Community Data Collection Effort","permalink":"/posts/contribute-to-xatu-data"}},"content":"We\'re excited to announce that Execution layer data, extracted via [cryo](https://github.com/paradigmxyz/cryo), is now available in the Xatu dataset.\\n\\nSummary:\\n\\n- The data is dedicated to the public domain under the CC BY 4.0 license\\n- Data is partitioned in 1000 block chunks in [Apache Parquet](https://parquet.apache.org) files\\n- We\'ve extracted data all the way back to genesis\\n\\nBy providing this data we hope to enable more in-depth analysis and research into the Ethereum ecosystem and compliment the rest of the dataset.\\n\\n## Tables\\n\\n- [canonical_execution_block](/data/xatu/datasets/canonical_execution_/#canonical_execution_block)\\n- [canonical_execution_transaction](/data/xatu/datasets/canonical_execution_/#canonical_execution_transaction)\\n- [canonical_execution_traces](/data/xatu/datasets/canonical_execution_/#canonical_execution_traces)\\n- [canonical_execution_logs](/data/xatu/datasets/canonical_execution_/#canonical_execution_logs)\\n- [canonical_execution_contracts](/data/xatu/datasets/canonical_execution_/#canonical_execution_contracts)\\n- [canonical_execution_four_byte_counts](/data/xatu/datasets/canonical_execution_/#canonical_execution_four_byte_counts)\\n- [canonical_execution_address_appearances](/data/xatu/datasets/canonical_execution_/#canonical_execution_address_appearances)\\n- [canonical_execution_balance_diffs](/data/xatu/datasets/canonical_execution_/#canonical_execution_balance_diffs)\\n- [canonical_execution_balance_reads](/data/xatu/datasets/canonical_execution_/#canonical_execution_balance_reads)\\n- [canonical_execution_erc20_transfers](/data/xatu/datasets/canonical_execution_/#canonical_execution_erc20_transfers)\\n- [canonical_execution_erc721_transfers](/data/xatu/datasets/canonical_execution_/#canonical_execution_erc721_transfers)\\n- [canonical_execution_native_transfers](/data/xatu/datasets/canonical_execution_/#canonical_execution_native_transfers)\\n- [canonical_execution_nonce_diffs](/data/xatu/datasets/canonical_execution_/#canonical_execution_nonce_diffs)\\n- [canonical_execution_nonce_reads](/data/xatu/datasets/canonical_execution_/#canonical_execution_nonce_reads)\\n- [canonical_execution_storage_diffs](/data/xatu/datasets/canonical_execution_/#canonical_execution_storage_diffs)\\n- [canonical_execution_storage_reads](/data/xatu/datasets/canonical_execution_/#canonical_execution_storage_reads)\\n\\n## About this data\\n\\n- Data is extracted via [cryo](https://github.com/paradigmxyz/cryo) only when the block is finalized on the beacon chain\\n- There is a delay before the [Apache Parquet](https://parquet.apache.org) files are available for public access of approximately ~6-24 hours\\n- Data is stored in chunks of **1000** blocks eg;\\n - `0.parquet` (block numbers **0** - **999**)\\n - `1000.parquet` (block numbers **1000** - **1999**)\\n - `2000.parquet` (block numbers **2000** - **2999**)\\n - `1000000.parquet` (block numbers **1000000** - **1000999**)\\n- All hex strings are lowercased, so remember to use the [`lower`](https://clickhouse.com/docs/en/sql-reference/functions/string-functions#lower) function eg. `WHERE transaction_hash = lower(\'0xABcd1234\')`\\n- For integer columns with a bit width greater than 64 (`Int128`/`UInt128`/`Int256`/`UInt256`) in parquet files, use the ClickHouse function [`reinterpretAsUIntX`](https://clickhouse.com/docs/en/sql-reference/functions/type-conversion-functions#reinterpretasuint256). For example, in the [canonical_execution_transaction](/data/xatu/datasets/canonical_execution_/#canonical_execution_transaction) table, the `value` column should be queried as `reinterpretAsUInt256(value)`.\\n\\n## Getting started\\n\\nThe easiest way to get started is to install the Clickhouse CLI and use `clickhouse local` to query the data. [Working with the data](/data/xatu/#working-with-the-data) has more details.\\n\\n### Examples\\n\\nThe following examples demonstrate some trivial queries you can run on the data.\\n\\n#### Example 1\\n\\
1nWhat are the top 5 block producers between block **20000000** and **20010999** on Mainnet?\\n\\n**Answer**: We can use the `canonical_execution_block` table to group on extra data column.\\n\\n```sql\\nclickhouse local -q \\"\\n SELECT\\n count(*) as count,\\n author,\\n extra_data_string\\n FROM\\n url(\'https://data.ethpandaops.io/xatu/mainnet/databases/default/canonical_execution_block/1000/{20000..20010}000.parquet\', \'Parquet\') \\n GROUP BY\\n author, extra_data_string\\n ORDER BY\\n count DESC\\n LIMIT 5\\n FORMAT PrettyCompact\\n\\"\\n\\n\u250c\u2500count\u2500\u252c\u2500author\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500extra_data_string\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\\n\u2502 5591 \u2502 0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5 \u2502 beaverbuild.org \u2502\\n\u2502 3389 \u2502 0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97 \u2502 Titan (titanbuilder.xyz) \u2502\\n\u2502 441 \u2502 0x1f9090aae28b8a3dceadf281b0f12828e676c326 \u2502 rsync-builder.xyz \u2502\\n\u2502 325 \u2502 0x1f9090aae28b8a3dceadf281b0f12828e676c326 \u2502 @rsyncbuilder \u2502\\n\u2502 220 \u2502 0xdf99a0839818b3f120ebac9b73f82b617dc6a555 \u2502 Illuminate Dmocratize Dstribute \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n```\\n\\n#### Example 2\\n\\nTop 5 transfers by value between blocks **20900000** and **20905999** on Mainnet?\\n\\n**Answer**: We can use the `canonical_execution_transaction` table to filter out transactions with no calldata and order by value.\\n\\n```sql\\nclickhouse local -q \\"\\n SELECT\\n block_number,\\n transaction_hash,\\n reinterpretAsUInt256(value) / 1e18 AS ether_value\\n FROM \\n url(\'https://data.ethpandaops.io/xatu/mainnet/databases/default/canonical_execution_transaction/1000/{20900..20905}000.parquet\', \'Parquet\') \\n WHERE \\n input IS NULL\\n ORDER BY \\n ether_value DESC\\n LIMIT 5\\n FORMAT PrettyCompact\\n\\"\\n\\n\u250c\u2500block_number\u2500\u252c\u2500transaction_hash\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500ether_value\u2500\u2510\\n\u2502 20902203 \u2502 0x62927f22026352b0af12fa844e588b999089849b98c50a89641cd19d69dfac35 \u2502 14008.381015295294 \u2502\\n\u2502 20903687 \u2502 0x39882cafc4195eee7b63444be4e9d16aad57e9465c2c4ef565758fa559a4e9f8 \u2502 8444.154971169999 \u2502\\n\u2502 20900096 \u2502 0x55a8950dc6e205bb1397bfe17a93767fb4ce026402b25ff9104476108b5d8807 \u2502 8357.538338940001 \u2502\\n\u2502 20902260 \u2502 0xbee9e1fcbc8405f6ef6620c08f47f568e8df0bb80feb5d77362aaf0e38e123a5 \u2502 7572.340613229151 \u2502\\n\u2502 20902737 \u2502 0x586f91964ed8a2adaab4f1c26a011b934a978c03e91303e9f74b206541fe9e21 \u2502 6800 \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n```\\n\\n#### Example 3\\n\\nWhat was the volume of USDT transferred between block **20900000** and **20905999** on Mainnet?\\n\\n**Answer**: We can use the `canonical_execution_erc20_transfers` table to sum the value of the USDT erc20 contract address.\\n\\n```sql\\nclickhouse local -q \\"\\n SELECT\\n sum(reinterpretAsUInt256(value)) / 1000000 as USDT\\n FROM\\n url(\'https://data.ethpandaops.io/xatu/mainnet/databases/default/canonical_execution_erc20_transfers/1000/{20900..20905}000.parquet\', \'Parquet\')\\n WHERE \\n erc20 = lower(\'0xdAC17F958D2ee523a2206206994597C13D831ec7\') -- USDT erc20 contract address\\n FORMAT PrettyCompact\\n\\"\\n\\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500USDT\u2500\u2510\\n\u2502 1585831125.724754 \u2502 -- 1.59 billion\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n```\\n\\n#### Example 4\\n\\nHow can I debug the transaction `0xe1cf349a8e55309cfab9d8d3dd8a8094765f99d65b20864c265bcd234abfe570` in block **20875205** that failed to execute?\\n\\n**Answer**: We can use the `canonical_execution_traces` table to get the full trace details. We know the transaction occured in block **20875205** so we can request the parquet chunk `20875000.parquet`.\\n\\n```sql\\nclickhouse local -q \\"\\n SELECT\\n trace_address,\\n action_from,\\n action_to,\\n action_value,\\n action_gas,\\n error\\n FROM\\n url(\'https://data.ethpandaops.io/xatu/mainnet/databases/default/canonical_execution_traces/1000/20875000.parquet\', \'Parquet\')\\n WHERE \\n transaction_hash = lower(\'0xe1cf349a8e55309cfab9d8d3dd8a8094765f99d65b20864c265bcd234abfe570\')\\n ORDER BY internal_index\\n FORMAT PrettyCompact\\n\\"\\n\\n\u250c\u2500trace_address\u2500\u252c\u2500action_from\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500action_to\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500action_value\u2500\u252c\u2500action_gas\u2500\u252c\u2500error\u2500\u2500\u2500\u2500\u2500\u2500\u2510\\n\u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502 0x1FCCC097DB89A86BFC474A1028F93958295B1FB7 \u2502 0xEEFBA1E63905EF1D7ACBA5A8513C70307C1CE441 \u2502 0 \u2502 29587220 \u2502 Reverted \u2502\\n\u2502 0 \u2502 0xEEFBA1E63905EF1D7ACBA5A8513C70307C1CE441 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0 \u2502 28883047 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 0_0 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x065347C1DD7A23AA043E3844B4D0746FF7715246 \u2502 0 \u2502 28421311 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 0_1 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x065347C1DD7A23AA043E3844B4D0746FF7715246 \u2502 0 \u2502 28405020 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 0_2 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x35A338522A435D46F77BE32C70E215B813D0E3AC \u2502 0 \u2502 28394712 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 0_3 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x35A338522A435D46F77BE32C70E215B813D0E3AC \u2502 0 \u2502 28391907 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 0_4 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x74C5A0
1D5DFCC6D4527C849F09ECC360C5345D986 \u2502 0 \u2502 28384423 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 0_4_0 \u2502 0x74C5A0D5DFCC6D4527C849F09ECC360C5345D986 \u2502 0xEFF8E65AC06D7FE70842A4D54959E8692D6AE064 \u2502 0 \u2502 27898562 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 0_4_0_0 \u2502 0xEFF8E65AC06D7FE70842A4D54959E8692D6AE064 \u2502 0x78910E1DFE6DF94EA7EEC54B25921673DB0E2A06 \u2502 0 \u2502 27457785 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n...\\n\u2502 221 \u2502 0xEEFBA1E63905EF1D7ACBA5A8513C70307C1CE441 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0 \u2502 67576 \u2502 Out of gas \u2502\\n\u2502 221_0 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x065347C1DD7A23AA043E3844B4D0746FF7715246 \u2502 0 \u2502 62480 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 221_1 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x065347C1DD7A23AA043E3844B4D0746FF7715246 \u2502 0 \u2502 48158 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 221_2 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x35A338522A435D46F77BE32C70E215B813D0E3AC \u2502 0 \u2502 45036 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 221_3 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x35A338522A435D46F77BE32C70E215B813D0E3AC \u2502 0 \u2502 44200 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 221_4 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x74C5A0D5DFCC6D4527C849F09ECC360C5345D986 \u2502 0 \u2502 41145 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 221_4_0 \u2502 0x74C5A0D5DFCC6D4527C849F09ECC360C5345D986 \u2502 0xEFF8E65AC06D7FE70842A4D54959E8692D6AE064 \u2502 0 \u2502 13996 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 221_4_0_0 \u2502 0xEFF8E65AC06D7FE70842A4D54959E8692D6AE064 \u2502 0x78910E1DFE6DF94EA7EEC54B25921673DB0E2A06 \u2502 0 \u2502 13345 \u2502 \u1d3a\u1d41\u1d38\u1d38 \u2502\\n\u2502 221_5 \u2502 0x9500F306CD233BF2BCFF1CC90D565C64626A162C \u2502 0x74C5A0D5DFCC6D4527C849F09ECC360C5345D986 \u2502 0 \u2502 6184 \u2502 Out of gas \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n```\\n\\n## Wrapping up\\n\\nWe hope you find this data useful and look forward to seeing what you build with it! If you have any questions or feedback please reach out to us on [Twitter](https://twitter.com/ethpandaops) or join the [Xatu Telegram Group](https://t.me/+JanoQFu_nO8yNzQ1)\\n\\nHappy querying!\\n\\nLove,\\n\\nEthPandaOps Team \u2764\ufe0f"},{"id":"contribute-to-xatu-data","metadata":{"permalink":"/posts/contribute-to-xatu-data","source":"@site/blog/2024-10-04-contribute-to-xatu-data/index.md","title":"Contribute to Xatu: Join the Community Data Collection Effort","description":"We\'re excited to announce that we are opening up the Xatu data collection pipeline to the Ethereum community! This initiative enables community members to contribute valuable data to the Xatu dataset.","date":"2024-10-04T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"consensus layer","permalink":"/posts/tags/consensus-layer"}],"readingTime":7.3,"hasTruncateMarker":false,"authors":[{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","t
1witter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null}],"frontMatter":{"slug":"contribute-to-xatu-data","title":"Contribute to Xatu: Join the Community Data Collection Effort","description":"We\'re excited to announce that we are opening up the Xatu data collection pipeline to the Ethereum community! This initiative enables community members to contribute valuable data to the Xatu dataset.","authors":["samcm","savid"],"tags":["xatu","consensus layer"],"image":"/img/blog/contribute-to-xatu-data.jpg","githubRepos":[{"name":"xatu","url":"https://github.com/ethpandaops/xatu"}],"relatedLinks":[{"name":"Contributoor Blog Post","url":"/posts/contributoor-beacon-node-companion"},{"name":"Xatu Lab Dashboard","url":"https://lab.ethpandaops.io/xatu"},{"name":"Working with Xatu Data","url":"/data/xatu/#working-with-the-data"},{"name":"Xatu Telegram Group","url":"https://t.me/+JanoQFu_nO8yNzQ1"}]},"unlisted":false,"prevItem":{"title":"Xatu Execution Layer data now available","permalink":"/posts/xatu-execution-layer"},"nextItem":{"title":"Xatu Consensus Layer P2P tables now available","permalink":"/posts/xatu-consensus-layer-p2p"}},"content":"import Alert from \'@site/src/components/Alert\';\\nimport Mermaid from \'@theme/Mermaid\';\\nimport { Details, Summary } from \'@site/src/components/Collapsible\';\\n\\n<Alert>\\nWe\'ve since released a new tool called [contributoor](https://github.com/ethpandaops/contributoor) that makes it easier to contribute to the Xatu dataset. [Check out the post](/posts/contributoor-beacon-node-companion/) for more information.\\n</Alert>\\n\\nWe\'re excited to announce that we are opening up the Xatu data collection pipeline to the Ethereum community! This initiative enables community members to contribute valuable data to the Xatu dataset.\\n\\nAs discussions regarding the potential increase in maximum blob count continue **we hope to shed light on the perspective of Ethereum\'s most crucial participants - home stakers.**\\n\\n**Summary:**\\n\\n- Privacy focused. Multiple privacy levels that allow contributors to only disclose data they\'re comfortable with\\n- Initially restricted to known community members\\n- [Data is published daily by the EthPandaOps team](/data/xatu/#working-with-the-data)\\n\\n## Data Collection\\n### Overview\\n\\nData is collected by running a Beacon node and the `xatu sentry` sidecar. The data is then sent to a pipeline that we run, which further anonymizes and redacts the data.\\n\\n<Mermaid\\n value={`graph TD\\n A1[Home Staker 1] --\x3e B1[Beacon Node]\\n A2[You!] --\x3e B2[Beacon Node]\\n A3[Home Staker 3] --\x3e B3[Beacon Node]\\n B1 --\x3e X1[Xatu Sentry]\\n B2 --\x3e X2[Xatu Sentry]\\n B3 --\x3e X3[Xatu Sentry]\\n C[EthPandaOps]\\n C --\x3e D[Data Pipeline]\\n \\n D --\x3e E[Public Parquet Files]\\n \\n X1 --\x3e C\\n X2 --\x3e C\\n X3 --\x3e C\\n\\n subgraph \\"Data Collection\\"\\n A1\\n A2\\n A3\\n B1\\n B2\\n B3\\n X1\\n X2\\n X3\\n end\\n \\n subgraph \\" \\"\\n C\\n D\\n end\\n \\n subgraph \\" \\"\\n E\\n end\\n linkStyle 0 stroke:#f66,stroke-width:2px;\\n linkStyle 1 stroke:#f66,stroke-width:2px;\\n linkStyle 2 stroke:#f66,stroke-width:2px;\\n linkStyle 3 stroke:#f66,stroke-width:2px;\\n linkStyle 4 stroke:#f66,stroke-width:2px;\\n linkStyle 5 stroke:#f66,stroke-width:2px;\\n linkStyle 6 stroke:#f66,stroke-width:2px;\\n linkStyle 7 stroke:#f66,stroke-width:2px;\\n linkStyle 8 stroke:#f66,stroke-width:2px;\\n linkStyle 9 stroke:#f66,stroke-width:2px;\\n linkStyle 10 stroke:#f66,stroke-width:2px;`}\\n/>\\n\\n### Events Collected\\n\\nThe following events will be collected:\\n\\n- **[beacon_api_eth_v1_events_head](/data/xatu/tables?table=beacon_api_eth_v1_events_head)**\\n - When the beacon node has a new head block\\n<Details>\\n<Summary>Example payload</Summary>\\n```yaml\\ndata:\\n block: \\"0x43d85cfa70181f60971dbc59d60c0e82e2ff8aea995bc942dc9c27bb16a055ca\\"\\n current_duty_depen
1dent_root: \\"0xc59a164bf477f138363db57e34f5b0e561c8bb1d30a0526f195b5575b2137513\\"\\n previous_duty_dependent_root: \\"0xbdbad239bcde3aa281edb7067a86ddba41f7f0a2e55b7ca61d628e57b6f1695f\\"\\n slot: \\"10098904\\"\\n state: \\"0xbcf7bbd9f5da8b88d09e3876834e93945edd98a258091339caedad2ec6764576\\"\\nevent:\\n date_time: \\"2024-10-04T03:01:13.245589039Z\\"\\n id: \\"b6b13f23-6412-4e74-aa62-8639fc2fa04e\\"\\n name: \\"BEACON_API_ETH_V1_EVENTS_HEAD_V2\\"\\nadditional_data:\\n epoch:\\n number: \\"315590\\"\\n start_date_time: \\"2024-10-04T02:56:23Z\\"\\n propagation:\\n slot_start_diff: \\"2245\\"\\n slot:\\n start_date_time: \\"2024-10-04T03:01:11Z\\"\\n```\\n</Details>\\n- **[beacon_api_eth_v1_events_block](/data/xatu/tables?table=beacon_api_eth_v1_events_block)**\\n - When the beacon node has a new block\\n<Details>\\n<Summary>Example payload</Summary>\\n```yaml\\ndata:\\n block: \\"0x7bb7f9e703896d516a0ee56d273dbe8fd71fd994a2f36cc489b8e1b825d74d44\\"\\n slot: \\"10098966\\"\\nevent:\\n date_time: \\"2024-10-04T03:13:37.703055591Z\\"\\n id: \\"58ccd540-81c2-44ce-820d-e73b5af0bea7\\"\\n name: \\"BEACON_API_ETH_V1_EVENTS_BLOCK_V2\\"\\nadditional_data:\\n epoch:\\n number: \\"315592\\"\\n start_date_time: \\"2024-10-04T03:09:11Z\\"\\n propagation:\\n slot_start_diff: \\"2703\\"\\n slot:\\n number: \\"10098966\\"\\n start_date_time: \\"2024-10-04T03:13:35Z\\"\\n```\\n</Details>\\n\\n- **[beacon_api_eth_v1_events_blob_sidecar](/data/xatu/tables?table=beacon_api_eth_v1_events_blob_sidecar)**\\n - When the beacon node has recieved a blob sidecar that passes gossip validation.\\n<Details>\\n<Summary>Example payload</Summary>\\n```yaml\\ndata:\\n block_root: \'0xc78adbc7ce7ab828bed85fedc6429989b4f4451d41aac8dc0c40b9f57839a3d7\'\\n index: \'0\'\\n kzg_commitment: \'0xa8de65da8d07703217d6879c75165a36973ff3ddace933907e7d400662b90e575812bb1302bfd4bb24691a550a0dc02a\'\\n slot: \'10099003\'\\n versioned_hash: \'0x0196e5bc26c289ff58a37c75f72b6824507d67ab0e43577495d1ad7b74716601\'\\nevent:\\n date_time: \'2024-10-04T03:21:00.752889196Z\'\\n id: adbf1ecb-4e52-404f-b3ba-6f83f6ffc4db\\n name: BEACON_API_ETH_V1_EVENTS_BLOB_SIDECAR\\nadditional_data:\\n epoch:\\n number: \'315593\'\\n start_date_time: \'2024-10-04T03:15:35Z\'\\n propagation:\\n slot_start_diff: \'1752\'\\n slot:\\n number: \'10099003\'\\n start_date_time: \'2024-10-04T03:20:59Z\'\\n```\\n</Details>\\n- **[beacon_api_eth_v1_events_chain_reorg](/data/xatu/tables?table=beacon_api_eth_v1_events_chain_reorg)**\\n - When the beacon node has reorganized its chain\\n<Details>\\n<Summary>Example payload</Summary>\\n```yaml\\ndata:\\n depth: \'3\'\\n epoch: \'83615\'\\n new_head_block: \'0x4a99bc2dbb2c5640cf0798102588dcbc3c02d15989c7652bbcf4647e24a14881\'\\n new_head_state: \'0x3e5af57c5c3bd8fa394c21edd8ac5b07378ef1e143ed18a9ff695090c970b23f\'\\n old_head_block: \'0x28e85b3e33721ad20b86c671f35686c8c91b5a29c6fd0cb41698872048d1b8ed\'\\n old_head_state: \'0x00f61794f1da3817bb8ae4591bbc0bc9cc0c72f4a422d5fdda5cd584ee147cd3\'\\n slot: \'2675702\'\\nevent:\\n date_time: \'2024-10-04T03:00:36.161478913Z\'\\n id: b0db9607-a862-4dd2-b7e6-4faf77e3a949\\n name: BEACON_API_ETH_V1_EVENTS_CHAIN_REORG_V2\\nadditional_data:\\n epoch:\\n number: \'83615\'\\n start_date_time: \'2024-10-04T02:56:00Z\'\\n propagation:\\n slot_start_diff: \'12161\'\\n slot:\\n start_date_time: \'2024-10-04T03:00:24Z\'\\n```\\n</Details>\\n- **[beacon_api_eth_v1_events_finalized_checkpoint](/data/xatu/tables?table=beacon_api_eth_v1_events_finalized_checkpoint)**\\n - When the beacon node\'s finalized checkpoint has been updated\\n<Details>\\n<Summary>Example payload</Summary>\\n```yaml\\ndata:\\n block: \'0x418645de30f82a71b7470dfc9831602f750a3b8e14e507e112791d53b3d3842e\'\\n epoch: \'188220\'\\n state: \'0x195dcdf004596c7afd999c39ff6f718f5bb631f3c8838b445fe87ea8f4f6de52\'\\nevent:\\n date_time: \'2024-10-04T03:00:47.506914227Z\'\\n id: 57e595a9-c79a-458c-be83-0d6dd58ee81c\\n name: BEACON_API_ETH_V1_EVENTS_FINALIZED_CHECKPOINT_V2\\nadditional_data:\\n epoch:\\n number: \'188220\'\\n start_date_time: \'2024-10-04T02:48:00Z\'\\n```\\n</Details>\\n\\n### Metadata\\n\\nThe following additional metadata is sent with every event:\\n#### Client Metadata\\n```yaml\\nclock_drift: \'2\' # Clock drift of the host machine\\nethereum:\\n consensus:\\n implementation: lighthouse # Beacon node implementation\\n version: Lighthouse/v5.3.0-d6ba8c3/x86_64-linux # Beacon node version\\n network:\\n id: \'11155111\' # Ethereum network ID\\n name: sepolia # Ethereum network name\\nid: 98df53c0-3de0-477c-a7c9-4ea9b17981c3 # Session ID. Resets on restart\\nimplementation: Xatu\\nmodule_name: SENTRY\\nname: b538bfd92sdv3 # Name of the sentry. Hash of the Beacon Node\'s node ID.\\nos: linux # Operating system of the host running sentry\\nversion: v0.0.202-3645eb8 # Xatu version\\n```\\n\\n#### Server Metadata\\nOnce we recieve the event, we do some additional processing to get the server metadata. The data that is added to the event is configurable per-user and allows users to only disclose data they\'re comfortable with. Geo location data is very useful for understanding how data is propagated through the network, but is not required.\\n\\n```yaml\\nserver:\\
1n client:\\n geo:\\n # OPTIONAL FIELDS\\n ## Data about ISP\\n autonomous_system_number: 24940 # Autonomous system number of the client\\n autonomous_system_organization: \\"Hetzner Online GmbH\\" # Organization associated with the autonomous system\\n \\n ## Data about location\\n city: \\"Helsinki\\" # City where the client is located\\n continent_code: \\"EU\\" # Continent code of the client\'s location\\n country: \\"Finland\\" # Country where the client is located\\n country_code: \\"FI\\" # Country code of the client\'s location\\n \\n ### ALWAYS REDACTED\\n latitude: REDACTED # Latitude coordinate of the client\'s location\\n longitude: REDACTED # Longitude coordinate of the client\'s location\\n group: \\"asn-city\\" # Group the client belongs to\\n user: \\"simplefrog47\\" # Pseudo username that sent the event\\n # ALWAYS REDACTED\\n ip: \\"REDACTED\\" # IP address of the client that sent the event\\n event:\\n received_date_time: \\"2024-10-04T03:00:48.533351629Z\\" # Timestamp when the event was received\\n```\\n\\n**Note:** \\n- The `client.name` field is re-hashed with a salt that only the EthPandaOps team has access to. This means that the original name of the client is not disclosed, and there is no way to map events back to a specific node id.\\n- The `client.ip`, `client.geo.latitude`, and `client.geo.longitude` fields are ALWAYS redacted.\\n\\n## Privacy groups\\n\\nPrivacy is a top priority for us. We have created privacy groups to allow users to only disclose data they\'re comfortable with.\\n\\n### No additional Geo/ASN data\\n<Details>\\n<Summary>No additional Geo/ASN data</Summary>\\n```yaml\\nautonomous_system_number: REDACTED # REDACTED\\nautonomous_system_organization: REDACTED # REDACTED\\ncity: \\"REDACTED\\" # REDACTED\\ncountry: \\"REDACTED\\" # REDACTED\\ncountry_code: \\"REDACTED\\" # REDACTED\\ncontinent_code: \\"REDACTED\\"\\n```\\n</Details>\\n\\n### With ASN data\\n<Details>\\n<Summary>Share geo location down to the city level</Summary>\\n```yaml\\nautonomous_system_number: 24940\\nautonomous_system_organization: \\"Hetzner Online GmbH\\"\\ncity: \\"Helsinki\\"\\ncontinent_code: \\"EU\\"\\ncountry: \\"Finland\\"\\ncountry_code: \\"FI\\"\\n```\\n</Details>\\n\\n<Details>\\n<Summary>Share geo location down to the country level</Summary>\\n```yaml\\nautonomous_system_number: 24940\\nautonomous_system_organization: \\"Hetzner Online GmbH\\"\\ncontinent_code: \\"EU\\"\\ncountry: \\"Finland\\"\\ncountry_code: \\"FI\\"\\ncity: \\"REDACTED\\" # REDACTED\\n```\\n</Details>\\n\\n<Details>\\n<Summary>Share geo location down to the continent level</Summary>\\n```yaml\\nautonomous_system_number: 24940\\nautonomous_system_organization: \\"Hetzner Online GmbH\\"\\ncontinent_code: \\"EU\\"\\ncity: \\"REDACTED\\" # REDACTED\\ncountry: \\"REDACTED\\" # REDACTED\\ncountry_code: \\"REDACTED\\" # REDACTED\\n```\\n</Details>\\n\\n<Details>\\n<Summary>Share no geo location data</Summary>\\n```yaml\\nautonomous_system_number: 24940\\nautonomous_system_organization: \\"Hetzner Online GmbH\\"\\ncontinent_code: \\"EU\\"\\ncity: \\"REDACTED\\" # REDACTED\\ncountry: \\"REDACTED\\" # REDACTED\\ncountry_code: \\"REDACTED\\" # REDACTED\\n```\\n</Details>\\n\\n### Without ASN data\\n<Details>\\n<Summary>Share geo location down to the city level without ASN</Summary>\\n```yaml\\ncity: \\"Helsinki\\"\\ncontinent_code: \\"EU\\"\\ncountry: \\"Finland\\"\\ncountry_code: \\"FI\\"\\nautonomous_system_number: REDACTED # REDACTED\\nautonomous_system_organization: REDACTED # REDACTED\\n```\\n</Details>\\n\\n<Details>\\n<Summary>Share geo location down to the country level without ASN</Summary>\\n```yaml\\ncontinent_code: \\"EU\\"\\ncountry: \\"Finland\\"\\ncountry_code: \\"FI\\"\\nautonomous_system_number: REDACTED # REDACTED\\nautonomous_system_organization: REDACTED # REDACTED\\ncity: \\"REDACTED\\" # REDACTED\\n```\\n</Details>\\n\\n<Details>\\n<Summary>Share geo location down to the continent level without ASN</Summary>\\n```yaml\\ncontinent_code: \\"EU\\"\\nautonomous_system_number: REDACTED # REDACTED\\nautonomous_system_organization: REDACTED # REDACTED\\ncity: \\"REDACTED\\" # REDACTED\\ncountry: \\"REDACTED\\" # REDACTED\\ncountry_code: \\"REDACTED\\" # REDACTED\\n```\\n</Details>\\n\\n## Get Started\\n\\nIf you\'re already running a beacon node, running `xatu sentry` is as simple as running a docker container on your node. For example:\\n```bash\\ndocker run -d --name xatu-sentry ethpandaops/xatu:latest \\\\\\n--preset ethpandaops \\\\\\n--beacon-node-url=http://localhost:5052 \\\\\\n--output-authorization=REDACTED\\n```\\n\\nContributing to the Xatu dataset is currently restricted to known community members. We have plans to open this up to the public in the future, but for now, we want to ensure that the data remains high quality and relevant to the home staker community (read: we need to make sure our pipeline can handle the increased load \ud83d\ude02)\\n\\nIf you\'d like to contribute to the Xatu dataset, please\\n**[apply for access here](/contribute-data/)**\\n\\
1nOnce you\'ve been granted access, you\'ll receive instructions on how exactly to run `xatu sentry` and start contributing to the dataset. Thank you!\\n\\n## Wrapping Up\\n\\nWe believe that by opening up the Xatu data collection pipeline we can empower the community to gain valuable insights and drive meaningful improvements to the Ethereum network. If you have any questions or feedback, please reach out to us on [Twitter](https://twitter.com/ethpandaops) or join the [Xatu Telegram Group](https://t.me/+JanoQFu_nO8yNzQ1).\\n\\nLove,\\n\\nEthPandaOps Team \u2764\ufe0f"},{"id":"xatu-consensus-layer-p2p","metadata":{"permalink":"/posts/xatu-consensus-layer-p2p","source":"@site/blog/2024-08-22-xatu-consensus-layer-p2p/index.md","title":"Xatu Consensus Layer P2P tables now available","description":"We\'re excited to announce that a handful of the Xatu Consensus Layer P2P tables are now publicly available in Parquet format!","date":"2024-08-22T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"consensus layer","permalink":"/posts/tags/consensus-layer"},{"inline":true,"label":"p2p","permalink":"/posts/tags/p-2-p"},{"inline":true,"label":"L1","permalink":"/posts/tags/l-1"}],"readingTime":4.59,"hasTruncateMarker":false,"authors":[{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"xatu-consensus-layer-p2p","title":"Xatu Consensus Layer P2P tables now available","authors":["savid","samcm"],"tags":["xatu","consensus layer","p2p","L1"],"image":"/img/blog/xatu-consensus-layer-p2p.jpg","githubRepos":[{"name":"xatu-data","url":"https://github.com/ethpandaops/xatu-data"},{"name":"probe-lab/hermes","url":"https://github.com/probe-lab/hermes"}],"relatedLinks":[{"name":"Apache Parquet","url":"https://parquet.apache.org"},{"name":"Xatu Telegram Group","url":"https://t.me/+JanoQFu_nO8yNzQ1"}]},"unlisted":false,"prevItem":{"title":"Contribute to Xatu: Join the Community Data Collection Effort","permalink":"/posts/contribute-to-xatu-data"},"nextItem":{"title":"Reusing the Kurtosis Ethereum-package as a base for your L2 devnet","permalink":"/posts/kurtosis-l2"}},"content":"import Alert from \'@site/src/components/Alert\';\\nimport Mermaid from \'@theme/Mermaid\';\\n\\nWe\'re excited to announce that a handful of the Xatu Consensus Layer P2P tables are now publicly available in Parquet format!\\n\\nAll data is captured directly from the Consensus Layer P2P network by utilising [Probelab\'s](https://probelab.io) [Hermes](https://github.com/probe-lab/hermes) project and [Xatu](https://github.com/ethpandaops/xatu). Instances are run by the EthPandaOps team in 2 regions.\\n\\nSummary:\\n\\n- The data is dedicated to the public domain under the CC BY 4.0 license\\n- Data is partitioned by hour or day in [Apache Parquet](https://parquet.apache.org) files\\n- We\'ve been collecting this data for the last few months!\\n\\nBy providing this data we hope to enable more in-depth analysis and research into the Ethereum network\'s behavior at the P2P level.\\n\\n## Tables\\n\\n- [libp2p_gossipsub_beacon_attestation](/data/xatu/datasets/libp2p_/#libp2p_gossipsub_beacon_attestation)\\n- [libp2p_gossipsub_beacon_block](/data/xatu/datasets/libp2p_/#libp2p_gossipsub_beacon_block)\\n- [libp2p_gossipsub_blob_sidecar](/data/xatu/datasets/libp2p_/#libp2p_gossipsub_blob_sidecar)\\n\\n<Alert>\\n**Note:** Data in these tables is only captured from a small subset of nodes and does not provide a complete picture of the network.\\n</Alert>\\n\\n## About this data\\n\\nThis data is captured closer to the wire in comparison to the those collected via the beacon api event stream. It is therefore not verified in any way. This means that messages with invali
1d signatures or other issues are **not** filtered out. **This allows for a more comprehensive view of the network, including potential malicious or faulty behavior that might be filtered out at higher levels, but sacrifices some data quality.**\\n\\nHere\'s a simplified diagram that shows how a block proposal would land in both the beacon api and the libp2p tables:\\n\\n<Mermaid\\n value={`graph TD\\n A[Beacon Node A] --\x3e|Block Proposal| B[P2P Network]\\n B --\x3e C[Beacon Node B]\\n C --\x3e D[Block Processing]\\n D --\x3e E[Beacon API Event Stream]\\n E --\x3e F[Xatu]\\n F --\x3e|Timestamped| I[beacon_api_eth_v1_events_block]\\n B --\x3e G[Xatu w/ Hermes]\\n G --\x3e|Timestamped| H[libp2p_gossipsub_beacon_block]\\n \\n style H fill:#FFFACD,stroke:#333,stroke-width:2px\\n style I fill:#FFFACD,stroke:#333,stroke-width:2px`}\\n/>\\n\\n## Getting started\\n\\nThe easiest way to get started is to install the Clickhouse client and use `clickhouse local` to query the data. [Working with the data](/data/xatu/#working-with-the-data) has more details.\\n\\n### Examples\\n\\nThe following examples demonstrate how these newly available tables can be utilized both independently and in conjunction with our existing Xatu tables to address intriguing questions about the Ethereum network.\\n\\n#### Example 1\\n\\nWhich 3 slots had the fastest arriving block proposals on 15th of August on Mainnet?\\n\\n**Answer**: We can use the `libp2p_gossipsub_beacon_block` table to find the first seen time for each slot and then order by that value.\\n\\n```sql\\nclickhouse local -q \\"\\n SELECT\\n min(propagation_slot_start_diff) as slot_time_ms,\\n slot \\n FROM \\n url(\'https://data.ethpandaops.io/xatu/mainnet/databases/default/libp2p_gossipsub_beacon_block/2024/8/15.parquet\', \'Parquet\') \\n GROUP BY\\n slot \\n ORDER BY\\n slot_time_ms\\n LIMIT 3\\n FORMAT PrettyCompact\\n\\"\\n\u250c\u2500slot_time_ms\u2500\u252c\u2500\u2500\u2500\u2500slot\u2500\u2510\\n\u2502 209 \u2502 9738557 \u2502\\n\u2502 229 \u2502 9742293 \u2502\\n\u2502 233 \u2502 9745006 \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n```\\n\\n#### Example 2\\n\\nWhich blocks were seen before the deadline but still didn\'t make it in to the finalized chain on Mainnet?\\n\\n**Answer**: We can check the `libp2p_gossipsub_beacon_block` table for blocks that were seen before the deadline on the p2p network, and then check the `canonical_beacon_block` table for blocks that were finalized.\\n\\n```sql\\nclickhouse local -q \\"\\n WITH p2p_blocks AS (\\n SELECT \\n block AS block_hash,\\n min(propagation_slot_start_diff) as first_seen_ms,\\n slot_start_date_time,\\n slot\\n FROM \\n url(\'https://data.ethpandaops.io/xatu/mainnet/databases/default/libp2p_gossipsub_beacon_block/2024/8/15.parquet\', \'Parquet\')\\n GROUP BY\\n block, slot_start_date_time, slot\\n HAVING\\n first_seen_ms < 4000\\n ),\\n canonical_counts AS (\\n SELECT \\n block_root AS block_hash, \\n COUNT(*) AS count\\n FROM \\n url(\'https://data.ethpandaops.io/xatu/mainnet/databases/default/canonical_beacon_block/2024/8/{14..16}.parquet\', \'Parquet\')\\n GROUP BY \\n block_root\\n ),\\n blocks_with_zero_count AS (\\n SELECT block_hash\\n FROM canonical_counts\\n WHERE count = 0\\n )\\n\\n SELECT \\n p2p.slot as slot,\\n p2p.first_seen_ms\\n FROM p2p_blocks p2p\\n LEFT JOIN canonical_counts c ON p2p.block_hash = c.block_hash\\n LEFT JOIN blocks_with_zero_count b ON p2p.block_hash = b.block_hash\\n WHERE COALESCE(c.count, 0) = 0\\n ORDER BY p2p.first_seen_ms ASC\\n LIMIT 10\\n FORMAT PrettyCompact\\n\\"\\n\u250c\u2500\u2500\u2500\u2500slot\u2500\u252c\u2500p2p.first_seen_ms\u2500\u2510\\n\u2502 9743140 \u2502 924 \u2502\\n\u2502 9743136 \u2502 3053 \u2502\\n\u2502 9738789 \u2502 3476 \u2502\\n\u2502 9738302 \u2502 3725 \u2502\\n\u2502 9743407 \u2502 3837 \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n```\\n\\n#### Example 3\\n\\nHow long did it take for attestations from validator 0 to be included on chain in the Sepolia network, and when were they first seen on the p2p network?\\n\\n**Answer:** We can compare the time attestations were seen on the Sepolia p2p network with the time they were included on chain. We\'ll use the `libp2p_gossipsub_beacon_attestation` table for p2p data and the `canonical_beacon_elaborated_attestation` table for when they were included on chain, joining them based on the block root.\\n\\nNote: Attestation files can be large so we\'re only looking at an hour of attestation data.\\n\\n```sql\\nclickhouse local -q \\"\\n WITH attestations AS (\\n SELECT\\n slot,\\n min(propagation_slot_start_diff) as first_seen_ms\\n FROM\\n url(\'https://data.ethpandaops.io/xatu/sepoli
1a/databases/default/libp2p_gossipsub_beacon_attestation/2024/8/8/0.parquet\', \'Parquet\')\\n WHERE attesting_validator_index = 0\\n GROUP BY slot\\n ),\\n on_chain_attestations AS (\\n SELECT\\n slot,\\n min(block_slot) as first_seen_slot\\n FROM\\n url(\'https://data.ethpandaops.io/xatu/sepolia/databases/default/canonical_beacon_elaborated_attestation/2024/8/8.parquet\', \'Parquet\')\\n WHERE has(validators, 0)\\n GROUP BY slot\\n )\\n SELECT \\n a.slot,\\n a.first_seen_ms as p2p_first_seen_ms,\\n o.first_seen_slot - a.slot AS inclusion_delay_slots\\n FROM attestations a\\n LEFT JOIN on_chain_attestations o ON a.slot = o.slot\\n ORDER BY a.slot ASC\\n FORMAT PrettyCompact\\n\\"\\n\u250c\u2500\u2500\u2500\u2500slot\u2500\u252c\u2500p2p_first_seen_ms\u2500\u252c\u2500inclusion_delay_slots\u2500\u2510\\n\u2502 5611821 \u2502 1036 \u2502 1 \u2502\\n\u2502 5611852 \u2502 885 \u2502 1 \u2502\\n\u2502 5611890 \u2502 4188 \u2502 1 \u2502\\n\u2502 5611920 \u2502 701 \u2502 1 \u2502\\n\u2502 5611960 \u2502 4145 \u2502 2 \u2502\\n\u2502 5611985 \u2502 580 \u2502 1 \u2502\\n\u2502 5612027 \u2502 4153 \u2502 2 \u2502\\n\u2502 5612037 \u2502 764 \u2502 1 \u2502\\n\u2502 5612067 \u2502 534 \u2502 1 \u2502\\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\\n```\\n\\n## Wrapping up\\n\\nWe hope you find this data useful and look forward to seeing what you build with it! If you have any questions or feedback please reach out to us on [Twitter](https://twitter.com/ethpandaops) or join the [Xatu Telegram Group](https://t.me/+JanoQFu_nO8yNzQ1)\\n\\nHappy querying!\\n\\nLove,\\n\\nEthPandaOps Team \u2764\ufe0f"},{"id":"kurtosis-l2","metadata":{"permalink":"/posts/kurtosis-l2","source":"@site/blog/2024-06-17-kurtosis-l2/index.md","title":"Reusing the Kurtosis Ethereum-package as a base for your L2 devnet","description":"Learn how to utilise the ethereum-package for the foundation of sophisticated tooling and L2 devnets.","date":"2024-06-17T00:00:00.000Z","tags":[{"inline":true,"label":"kurtosis","permalink":"/posts/tags/kurtosis"},{"inline":true,"label":"ethereum-package","permalink":"/posts/tags/ethereum-package"},{"inline":true,"label":"optimism-package","permalink":"/posts/tags/optimism-package"},{"inline":true,"label":"L1","permalink":"/posts/tags/l-1"},{"inline":true,"label":"L2","permalink":"/posts/tags/l-2"},{"inline":true,"label":"testnet","permalink":"/posts/tags/testnet"},{"inline":true,"label":"devnet","permalink":"/posts/tags/devnet"}],"readingTime":6.92,"hasTruncateMarker":false,"authors":[{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null},{"name":"barnabasbusa","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/barnabasbusa","github":"https://github.com/barnabasbusa","twitter":"https://x.com/BarnabasBusa","imageURL":"/img/team/barnabasbusa.jpeg","key":"barnabasbusa","page":null}],"frontMatter":{"slug":"kurtosis-l2","title":"Reusing the Kurtosis Ethereum-package as a base for your L2 devnet","authors":["parithosh","barnabasbusa"],"description":"Learn how to utilise the ethereum-package for the foundation of sophisticated tooling and L2 devnets.","tags":["kurtosis","ethereum-package","o
1ptimism-package","L1","L2","testnet","devnet"],"image":"/img/blog/kurtosis-l2.jpg","githubRepos":[{"name":"ethereum-package","url":"https://github.com/ethpandaops/ethereum-package"},{"name":"optimism-package","url":"https://github.com/ethpandaops/optimism-package"}],"relatedLinks":[{"name":"Kurtosis Deep Dive","url":"https://ethpandaops.io/posts/kurtosis-deep-dive/"},{"name":"Starlark","url":"https://github.com/bazelbuild/starlark"}]},"unlisted":false,"prevItem":{"title":"Xatu Consensus Layer P2P tables now available","permalink":"/posts/xatu-consensus-layer-p2p"},"nextItem":{"title":"Kurtosis: A Deep Dive to Local Devnets","permalink":"/posts/kurtosis-deep-dive"}},"content":"## Introduction\\nIn our latest series, which started with a [deep dive into Kurtosis](https://ethpandaops.io/posts/kurtosis-deep-dive/), we\'ve explored the utility of creating localized Ethereum devnets. Today, we expand on this by demonstrating how the [ethereum-package](https://github.com/ethpandaops/ethereum-package) can serve as the foundation for sophisticated tooling as well as L2 devnets.\\n \\nThe advantage of a package based approach on Kurtosis, is that packages can be imported and linked in other packages. This means that one can create a new package that imports the ethereum-package and expect that a base layer Ethereum will always be spun up with any minimal maintenance effort. This feature would be especially useful for anyone creating tooling that isn\'t strictly focussed on the Ethereum base layer, but still relies on it for data or state access. In order to expand on how this feature could be used, we decided on building an example that runs a L2 devnet but with the base layer being Ethereum, imported as a package. For this example, we will use the [Optimism](https://
1optimism.io) stack and create the [optimism-package](https://github.com/ethpandaops/optimism-package) purely due to the fact that it reuses many base layer components we already had definitions for. There is however no reason that the example could not apply to any other L2 or tool that relies on Ethereum as a base layer.\\n\\n## Interacting with imported packages in Kurtosis\\nKurtosis employs [Starlark](https://github.com/bazelbuild/starlark) for its scripting needs, allowing you to treat each folder as a module that can be conveniently imported into other modules, enhancing modularity and reuse. Let\'s take an example of two modules, one that defines the constants (defined in `constants.star`) and another that uses these constants (defined in `main.star`).\\n\\nThe `main.star` file can then import the constants defined in `constants.star` as follows:\\n```python\\nconstants = import_module(\\"../constants/constants.star\\")\\n```\\n\\nIn the same way that we imported a file from a module into another module, we can also import an entire package from github into any arbitrary module. This would look like this:\\n```python\\nethereum_package = import_module(\\"github.com/ethpandaops/ethereum-package/main.star\\")\\n```\\n\\nThe `ethereum-package` contains a `run` function that needs to be called with the arguments used to start up the Ethereum devnet. The response is stored in a variable called `l1`, this variable contains the context of the Ethereum devnet. We can then reference any variable stored in this context, for e.g we can obtain the RPC URL of an Ethereum node in order to interact with the underlying chain. The code to do so would look like this: \\n```python\\n# Run the Ethereum devnet and store the context in a variable called l1\\nl1 = ethereum_package.run(plan, ethereum_args)\\n# Read the RPC URL of the first participant in the Ethereum devnet and store it in a variable called l1_rpc\\nl1_rpc = l1.all_participants[0].el_context.rpc_http_url\\n```\\n\\nWe additionally contain prefunded accounts in the `ethereum-package` that can be used for transactions on the base chain. These funds can be accessed by calling the `pre_funded_accounts` in the `ethereum-package` context. We do reserve certain accounts for specific purposes, for example, the 12th account is reserved for the L2 contract deployer. A full list of pre-allocations can be found [here](https://github.com/ethpandaops/ethereum-package?tab=readme-ov-file#pre-funded-accounts-at-genesis). The code to access the private key of the 12th account would look like this:\\n```python\\nl1_priv_key = l1.pre_funded_accounts[12].private_key # reserved for L2 contract deployer\\n```\\nThe Ethereum RPC as well as a private key with funds would be integral components for building any tooling that interacts with the Ethereum base layer, in the case of the `optimism-package`, we pass these values on to the optimism contract deployer as well as the op-nodes that are spun up later in the process. A full example of how we\'ve interacted with the base `ethereum-package` can be found in this [main.star file](https://github.com/ethpandaops/optimism-package/blob/main/main.star). \\n\\n## Optimism package overview\\nThe optimism docs contain a page on all the requirements to create a L2 rollup testnet, the docs can be found [here](https://docs.optimism.io/builders/chain-operators/tutorials/create-l2-rollup). At a high level, the `optimism-package` deploys the following:\\n- Smart contracts\\n- Sequencer node (consensus and execution client)\\n- Batcher\\n- Proposer\\n\\nOnce we imported the `ethereum-package` and possess the L1 context, we pass on the required variables such as a prefunded privatekey, RPC, chainID and a few other values to the `contract_deployer.star` module. This module contains all the logic for deploying and configuring the optimism contracts. The `contract_deployer` module waits until the L1 in finalized to avoid issues and then proceeds with using `cast` and `forge` to deploy the contracts and using the `op-node genesis` command to create the L2 genesis files. \\n\\nThe L2 chain genesis files are then available in the `contract_deployer` context, accessible via `op_genesis.files_artifacts[0]` for use by the nodes. The package then continues on to launching the L2 network participants, this logic is encapsulated in the module `participant_network.star`. The module launches all the participants in the L2 network, including the sequencer, batcher, proposer and the op-nodes. If there are already setup nodes on the L2 network, then the module will connect the new nodes to the existing nodes. \\n\\nOnce the nodes, batcher and proposer are setup, the `optimism-package` will then turn to
1setting up tooling. This tooling currently includes just `op-blockscout`, but can be expanded to include any other tooling that is required. The `op-blockscout` is a fork of the original blockscout that is configured to work with the L2 network. The `op-blockscout` is another example of how extensible the module approach is, it references the L1 RPC from the `ethereum-package` as well as the L2 RPC from the `optimism-package` to provide a seamless experience for the developer. \\n\\nThe result of the `optimism-package` execution is a fully functional L2 devnet as well as a list of services and URLs to access them by.\\n\\n## Configuration examples\\nAs the `optimism-package` is designed to fully encompass the ethereum-package, its configuration file seamlessly integrates all parameters from the `ethereum-package` alongside its own unique settings. To differentiate from using the original `ethereum-package` a new field is introduced in the configuration file called `optimism_package`. This field will contain all the new configuration parameters that are specific to the L2 devnet.\\n\\n```yaml\\noptimism_package: # parameters specific to the optimism package\\n participants:\\n - el_type: op-geth\\n cl_type: op-node\\n additional_services:\\n - blockscout\\nethereum_package: # inherited from the ethereum package\\n participants:\\n - el_type: geth\\n - el_type: reth\\n network_params:\\n preset: minimal\\n additional_services:\\n - dora\\n - blockscout\\n```\\n\\n## What are the benefits of using Kurtosis for tooling and L2 devnets?\\nThe current way of creating any L2 devnet/tooling is to use an existing L1 testnet and deploy the L2 package/tooling on top of it or to use custom bash scripts to perform local testing. This is a cumbersome process and still requires a considerable amount of maintenance effort to ensure the L1 works as expected through upgrades. The Kurtosis approach allows you to create a local L1 devnet in a few minutes with relatively little maintenance effort and allows you to spend more time on the L2 devnet/tooling - enabling faster and safer prototyping. Kurtosis also works under the concept of [enclaves](https://docs.kurtosis.com/advanced-concepts/enclaves/), these enclaves are fully isolated from each other - allowing multiple tests to run in parallel without interference or networking issues.\\n\\n## What are the disadvantages of using Kurtosis for L2 devnets?\\nThe main disadvantage of using Kurtosis is the local nature of the devnet. This means that providing devnet access to other developers is not as easy as using a public testnet, this is a fundamental bottleneck in how kurtosis works today. Additionally, the lack of default persistent storage in Kurtosis implies that you lose all your data when you stop the devnet(unless saved ahead of time). This is not a problem for testing purposes, but it is something to keep in mind.\\n\\n## Conclusion\\nTo fully appreciate the power and flexibility of using Kurtosis with `ethereum-package`, we encourage you to initiate your own projects reusing it for tooling or to define your own L2. Explore the detailed documentation, experiment with the configurations, and join the discord to share your experiences!"},{"id":"kurtosis-deep-dive","metadata":{"permalink":"/posts/kurtosis-deep-dive","source":"@site/blog/2024-04-19-kurtosis-deep-dive/index.md","title":"Kurtosis: A Deep Dive to Local Devnets","description":"Kurtosis is tool to run ephemeral packages that are defined in Starlark which opens up a world of possibilities for developers to test their applications, quickly iterate and retest.","date":"2024-04-19T00:00:00.000Z","tags":[{"inline":true,"label":"kurtosis","permalink":"/posts/tags/kurtosis"},{"inline":true,"label":"ethereum-package","permalink":"/posts/tags/ethereum-package"},{"inline":true,"label":"devnet","permalink":"/posts/tags/devnet"},{"inline":true,"label":"testnet","permalink":"/posts/tags/testnet"},{"inline":true,"label":"shadowfork","permalink":"/posts/tags/shadowfork"}],"readingTime":16.76,"hasTruncateMarker":false,"authors":[{"name":"barnabasbusa","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/barnabasbusa","github":"https://github.com/barnabasbusa","twitter":"https://x.com/BarnabasBus
1a","imageURL":"/img/team/barnabasbusa.jpeg","key":"barnabasbusa","page":null},{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null}],"frontMatter":{"slug":"kurtosis-deep-dive","title":"Kurtosis: A Deep Dive to Local Devnets","description":"Kurtosis is tool to run ephemeral packages that are defined in Starlark which opens up a world of possibilities for developers to test their applications, quickly iterate and retest.","authors":["barnabasbusa","parithosh"],"tags":["kurtosis","ethereum-package","devnet","testnet","shadowfork"],"image":"/img/blog/kurtosis-deep-dive.jpg","githubRepos":[{"name":"ethereum-package","url":"https://github.com/ethpandaops/ethereum-package"}],"relatedLinks":[{"name":"Kurtosis","url":"https://www.kurtosis.com"}]},"unlisted":false,"prevItem":{"title":"Reusing the Kurtosis Ethereum-package as a base for your L2 devnet","permalink":"/posts/kurtosis-l2"},"nextItem":{"title":"Assertoor: Ethereum Testnet Testing Tool","permalink":"/posts/assertoor-introduction"}},"content":"import { Details, Summary } from \'@site/src/components/Collapsible\';\\n\\n## Introduction to Kurtosis\\n[Kurtosis](https://www.kurtosis.com) is tool to run ephemeral packages that are defined in [Starlark](https://github.com/bazelbuild/starlark) which opens up a world of possibilities for developers to test their applications, quickly iterate and retest. It is built from the ground up to be a developer-friendly platform that allows developers to test their applications in a controlled environment without the need for complex setup or configuration.\\n\\nThe major focus of this blog post is going to be a deep dive into the Kurtosis platform, exploring its features, capabilities, and how it can be used to test applications in a controlled environment. Primary focus will be on a package that is built on top of Kurtosis, called [ethereum-package](https://github.com/ethpandaops/ethereum-package). This package is the brainchild between the [Kurtosis](https://www.kurtosis.com/company) and the [ethPandaOps](https://ethpandaops.io/team/) teams, and is designed to make it easy for developers to test their Ethereum clients and corresponding tooling.\\n\\n\\n## Why would you need Kurtosis in the first place?\\nYou may have read that we\'ve often referenced the kurtosis/ethereum-package in our past presentations, tweets and blogposts. The reason we referenced it in many different places is because our team has been relying on it for rapid testing of various client releases. It is being used as the primary testbed for testing all of our tools as well. As a result of its deep integration with tools and its simplicity, Kurtosis has become our go-to approach for rapid iterative local testing.\\n\\nKurtosis works on the basis of overrides, all the configurable values can be overriden, allowing devs to easily swap a stable image for on they wish to test. This means we can perform integration testing on a local environment and quickly know if it breaks.\\n\\nAside from being useful as a standalone tool, Kurtosis can also be run on the CI to ensure that clients or tools are able to test their pull requests and ensure healthy releases. We\'ve packaged kurtosis along with our test assertion tool called assertoor into a github action called [kurtosis-assertoor-github-action](https://github.com/marketplace/actions/kurtosis-assertoor-github-action). Please do provide feedback on kurtosis and assertoor if we can help make it easier to run in your CI pipelines.\\n\\n### Local devnets for the masses\\nThe standard way to launch devnets has traditionally been via bespoke scripts. These scripts often were tailor made for a specific client and did not include any tooling. Launching publicly accessible devnets took a lot more organization and collaboration, often requiring the help of a DevOps engineer. Kurtosis makes it possible to reduce this toil and has automated the process of setting up private devnets in a matter of minutes, it extends this devnet with tooling as well - enabling a whole new range of developers to perform local development.\\n\\n## How to use Kurtosis (basic guide)\\n\\n#### Pre-requisites\\n- [Docker](https://github.com/docker/docker-install?tab=readme-ov-file#usage)\\n- [Kurtosis](https://docs.kurtosis.com/install)\\n\\n#### Concepts\\nKurtosis is a CLI tool that c
1onnects to the Kurtosis engine, which is a server that can then interface with the system to launch a package. It is important that the kurtosis engine is always running before you run any package.\\n\\nThe Kurtosis engine utilizes the `docker` or `kubernetes` backend to execute the defined package. The `docker` mode would imply that the code is executed on a single instance, whereas the `kubernetes` mode would imply that the code is executed on a multi-node kubernetes cluster. This means Kurtosis packages can benefit from local execution as well as scalable remote execution.\\n\\nKurtosis is built around the concept of [enclaves](https://docs.kurtosis.com/advanced-concepts/enclaves/). These enclaves are a way to separate executions, similar to `namespace` in kubernetes. The participants of one enclave usually cannot interact in any way with the participants of another enclave.\\n\\n\\n#### Quick start guide\\nTo get started with your first devnet on your local machine, you can run the following command:\\n```bash\\nkurtosis engine start\\n```\\n\\nWe can then use the `kurtosis run <package>` to start any package. This package is a collection of [starlark](https://github.com/bazelbuild/starlark) code that kurtosis can interpret. The package can be referenced via the Github URL or the local path of the git repository. You can explore all the flags by running:\\n\\n```bash\\nkurtosis run --help\\n```\\n\\nFor our purposes, we will be using the ethereum-package. To run the ethereum-package, you can run the following command:\\n```\\nkurtosis run github.com/ethpandaops/ethereum-package\\n```\\n\\nThis will spin up the default Ethereum devnet with a single node (geth/lighthouse pair), kurtosis takes care of creating the genesis state, peering, validator keys and ensuring the nodes are spun up with the right flags. We will explore how to customize this to suit your needs later in the post.\\n\\nThe output of the run command will be a list of participants as well as the ports they have exposed. You can use these ports to interact with the devnet as you would on any other network.\\n\\nIn order to clean up all the resources that were instantiated by the `kurtosis run` command, you can run the following:\\n```bash\\nkurtosis enclave rm <enclave-name> -f\\n```\\n\\nIf no other enclaves are running, you can also run the following command to clean up all the resources:\\n```bash\\nkurtosis clean -a\\n```\\n\\n#### Customizing the devnet\\nKurtosis runs on the basis of overriding parameters, hence if the default run parameters are insufficient, you can run kurtosis with `--args-file <path-to-args-file>`. This flag allows you to pass in a YAML/JSON file with all the parameters you want to customize, when a parameter is not configured then its default value is utilized. An exhaustive list of all possible configuration parameters can be found [here](https://github.com/ethpandaops/ethereum-package?tab=readme-ov-file#configuration).\\n\\nWe maintain a non-exhaustive list of completed configuration files that can be found in the [tests folder of the ethereum-package repository](https://github.com/ethpandaops/ethereum-package/tree/main/.github/tests). We hope that these examples can showcase how powerful the package can be.\\n\\nYou can also reference yaml files that are hosted online by using the raw URL of the given file. Here is an example of how you can run a devnet with a custom configuration file:\\n```bash\\nkurtosis run github.com/ethpandaops/ethereum-package --args-file https://raw.githubusercontent.com/ethpandaops/ethereum-package/main/.github/tests/mix.yaml\\n```\\n\\n\\n#### Customizing the different participants\\nParticipants in a network refer to the nodes that will partake in the devnet. You can specify multiple clients of the same configuration with `count` parameter. You can specify differing participants by using a new element. Here is an example:\\n```yaml\\nparticipants:\\n - el_type: geth\\n cl_type: lighthouse\\n count: 3\\n - el_type: nethermind\\n cl_type: teku\\n```\\nThe above example would spin up 3x geth/lighthouse nodes and 1x nethermind/teku with their default parameters. They will all get an equal number of validators and would be provided the same genesis files as well as peering information.\\n\\n#### Overriding the default configuration\\nYou can specify a parameter in the config file in order to override it, the same applies for client images. Here is an example of a configuration file that uses a custom docker
1container for the nethermind node:\\n```yaml\\nparticipants:\\n - el_type: nethermind\\n el_image: ethpandaops/nethermind:best-image\\n - el_type: besu\\n cl_type: lighthouse\\n```\\nIn this example, we will spin up a network with nethermind/lighthouse(default CL) and a besu/lighthouse node. Since the nethermind node has a custom image, it will be spun up with the specified image and the others will use the default image.\\n\\n\\n#### Run custom number of validators, with a different validator client\\nIn order to run a client pair with a custom number of validators using a different validator client type you can use the following configuration file:\\n```yaml\\nparticipants:\\n - el_type: geth\\n cl_type: lighthouse\\n vc_type: lodestar\\n validator_count: 100\\n```\\nIn the above example you will spin up a devnet with 1 geth node, 1 lighthouse node, and 100 lodestar validator nodes, that are connected to the lighthouse node.\\n\\n#### Extra parameters, environment variables, and additional labels\\nIf you would like to pass additional flags, parameters, environment variables, or labels to the docker containers that are being spun up, you can use the following configuration file (also possible for `cl` and `vc`):\\n```yaml\\nparticipants:\\n - el_type: geth\\n cl_type: lighthouse\\n count: 3\\n el_extra_params:\\n - \\"--rpc\\"\\n el_extra_env_vars: {\\"ETH_RPC_PORT\\": \\"8545\\"}\\n el_extra_labels: {\\"com.example.label\\": \\"example\\"}\\n```\\n\\nUsing this information you should be able to have a basic understanding of what you can do with the ethereum-package and how you can customize the participants to your needs.\\n\\n#### Deploying the entire MEV stack\\nKurtosis supports the entire MEV as well as MEV testing stack. This includes spinning up mev-boost, mev-relays, builders and the frontend to showcase delivered payloads. Here is an example of how you can run a devnet with the entire MEV stack:\\n```yaml\\nparticipants:\\n- el_type: geth\\n cl_type: lighthouse\\n count: 3\\nmev_type: full\\n```\\n\\nThe `mev_type` full would deploy the entire MEV stack as seen on Mainnet ethereum. Each portion of the MEV stack can be customized for testing purposes as well. However, for easier client side protyping we also support `mev_type: mock`, which deploys a mock implementation of the relay stack and is managed by the tool [here](https://github.com/marioevz/mock-builder/). This `mock` mode can help iterate on builder API changes without needing to update many components.\\n\\nBy customizing the `mev_boost_image` config value, we can also utilize this workflow to test tools that operate at the layer of mev-boost.\\n\\n## Customize the network parameters (basic)\\nThe next step is to dive into the network parameters that you can customize. These parameters are defined in the `network_params` section of the configuration file.\\n\\nThe most commonly used parameters are `seconds_per_slot`, `genesis_delay` and `electra_fork_epoch`. These parameters help your network to produce slots faster or slower, and genesis_delay will help you to delay the genesis block of the network. The `electra_fork_epoch` parameter will help you to define when the electra fork should happen. This can help you test out feature branches by activating the next fork at any given epoch. Here is an example of how you can define these parameters in the configuration file:\\n```yaml\\nnetwork_params:\\n seconds_per_slot: 6\\n genesis_delay: 120\\n electra_fork_epoch: 5\\n```\\n\\nMore advanced configuration options such as `min_validator_withdrawability_delay` and `shard_committee_period` can also be adjusted.\\n\\n## Customize the network parameters (advanced)\\n\\n#### Minimal preset\\nIf you would like to use a a [`minimal preset`](https://github.com/ethereum/consensus-specs/tree/dev/presets/minimal) for the network you can use the following configuration file:\\n```yaml\\nnetwork_params:\\n preset: minimal\\n```\\nHowever, keep in mind that this preset requires custom consensus layer clients to be defined in the configuration file, an example for these client pairs can be found [here](https://github.com/ethpandaops/ethereum-package/blob/main/.github/tests/minimal.yaml). `minimal` preset can also break some tooling, so please proceed with caution.\\n\\n#### Shadowfork and bootstrapping (advanced)\\nThe ethereum-package now supports bootstrapping and shadowforks by defining the `network` and `network_sync_base_url` parameters in the configuration file. This will help you to bootstrap the network from a specific snapshot, and to run a shadowfork of the network. Here is an example of how you can define these parameters in the configuration file:\\n\\n```yaml\\nnetwork_params:\\n network: holesky-shadowfork\\n network_sync_base_url: https://ethpandaops-ethereum-node-snapshots.ams3.digitaloceanspaces.com/\\npersistent: true\\n```\\nThese parameters will pull a latest snapshot from holesky network as defined in the s3 bucket linked above. This snapshot was created with the tool [snapshotter](https://github.com/ethpandaops/snapsh
1otter), and specifies the format to store the data in. Once the data has been imported, The genesis tools will create the requisite data for the clients to start a new beaconchain and perform shadowfork at the defined `electra_fork_epoch`.\\n\\nPersistence is highly recommended when using shadowforks, as this requires multiple gigabytes of storage to store the chain data, and storing it all in RAM or SWAP is not recommended.\\n\\nNote: `kurtosis clean -a` will wipe the chain data and nuke the enclave, and you will have to re-download the snapshot if you want to run the shadowfork again.\\n\\n## How to use Kurtosis (advanced guide)\\nSo at this point you should be fairly familiar with how the ethereum-package is used, and you should be fairly familiar with how to customize the devnet to your needs. In this section we are going to explore some of the more advanced features of the ethereum-package and how you can use them to your advantage.\\n\\n#### Long running devnets using Kubernetes\\nIf you are looking to run a devnet for a longer period of time, it is advisable to use Kubernetes as the backend engine of kurtosis. This requires a more verbose. A pre-requisite is that you have a Kubernetes cluster running, and that cluster is set as your current context. (The setup of a kubernetes cluster or how to set the context is out of the scope of this blog post, but you can find more information [here](https://kubernetes.io/docs/setup/)).\\n\\nAdjust your `kurtosis-config.yaml` where you define the connection parameters of the kurtosis engine using the following [guide](https://docs.kurtosis.com/k8s/#iii-add-your-cluster-information-to-kurtosis-configyml).\\n\\nAfter you have set up your kurtosis-config.yaml file, you can start the kurtosis engine with the following command:\\n```bash\\nkurtosis cluster set <cluster-name>\\nkurtosis engine restart\\nkurtosis gateway\\n```\\n\\nNote: The `kurtosis gateway` command will start a local proxy that will allow you to connect to the kurtosis engine running on the Kubernetes cluster. This is necessary to run in the background whenever you are interacting with the kurtosis engine. If the `gateway` command is not running, then you will not be able to interact with the kurtosis engine but it will not affect any running enclaves.\\n\\nThen from this point on you can use the same `kurtosis run` command to spin up your devnet, which will target the Kubernetes cluster instead of the local machine. This enables you to run devnets for a longer period of time, and you can even share the devnet with your team members. Also this enables you to run much larger devnets where your (automatically scaling - optionally) Kubernetes cluster can handle the load.\\n\\nComplete guide on all the possible configurable parameters can be found on the very detailed documentation of the [ethereum-package](https://github.com/ethpandaops/ethereum-package?tab=readme-ov-file#configuration).\\n\\n#### Kubernetes specific configuration flags\\nWhen running the devnet on Kubernetes, there are some additional flags that you can use to customize the devnet to your needs. A few of these flags are [tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) and [node selectors](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector). These flags can be used to define which nodes the devnet should run on, and which nodes the devnet should not run on. These flags are highly recommended to be defined when using a kubernetes cluster that is shared with other workloads. Node selectors and tolerations can help you ensure that the devnet is running on a node that is not being used by other workloads, and that the devnet is not interfer
1ing with other workloads.\\n\\nThese flags are:\\n```yaml\\nparticipants:\\n - el_type: geth\\n cl_type: lighthouse\\n count: 3\\n node_selectors:\\n kubernetes.io/hostname: node1\\n tolerations: # tolerations for this participant\\n - key: \\"key\\"\\n cl_toleration: [] # tolerations for the consensus client\\n el_toleration: [] # tolerations for the execution client\\n vc_toleration: [] # tolerations for the validator client\\n ```\\n\\nAnother important flag that can be defined are CPU and memory requests and limits. Each client type has a default value for these parameters, but you can override them by defining them in the configuration file. These flags are (also available for `cl` and `vc`):\\n```yaml\\nel_min_cpu: 0\\nel_max_cpu: 0\\nel_min_mem: 0\\nel_max_mem: 0\\nel_volume_size: 0\\n```\\nCustomizing these values will help you ensure that the devnet is running on a node that has enough resources to handle the load, and assist you in avoiding any resource contention with other workloads that are running on the same node. As well as help your kubernetes cluster to automatically scale up/down based on the resource requirements of the devnet.\\n\\n\\n## Conclusion\\nIn this blog post we have explored the basic and advanced features of the ethereum-package, and how you can use them to your advantage. We have explored how you can customize different participants, devnet network parameters to your needs, and how you can run the devnet on docker or Kubernetes. We hope that this blog post has given you a good understanding of how you can use the ethereum-package to test your Ethereum clients and tooling in a controlled environment.\\n\\nA massive thanks goes out to the whole Kurtosis team for building such an amazing platform! \ud83d\ude80\\n\\n## References\\n### Ethereum-package clients and tooling\\nThere is a wide range of ethereum clients that are supported out of the box by the ethereum-package. These clients are:\\n\\n<Details>\\n<Summary><strong>Execution Layer Clients</strong></Summary>\\n\\n- [Geth](https://geth.ethereum.org) - Go implementation of the Ethereum protocol\\n- [Nethermind](https://www.nethermind.io) - High-performance .NET Core implementation\\n- [Besu](https://besu.hyperledger.org) - Java implementation by Hyperledger\\n- [Erigon](https://github.com/ledgerwatch/erigon) - Implementation focused on efficiency\\n- [Reth](https://github.com/paradigmxyz/reth) - Rust implementation by Paradigm\\n- [Nimbus-eth1](https://github.com/status-im/nimbus-eth1) - Nim implementation\\n</Details>\\n\\n<Details>\\n<Summary><strong>Consensus Layer Clients</strong></Summary>\\n\\n- [Lighthouse](https://lighthouse-book.sigmaprime.io) - Rust implementation by Sigma Prime\\n- [Teku](https://consensys.io/teku) - Java implementation by ConsenSys\\n- [Nimbus-eth2](https://nimbus.guide) - Nim implementation for resource-restricted devices\\n- [Prysm](https://prysmaticlabs.com) - Go implementation by Prysmatic Labs\\n- [Lodestar](https://lodestar.chainsafe.io) - TypeScript implementation by ChainSafe\\n- [Grandine](https://github.com/grandinetech/grandine) - Rust implementation by Grandine\\n</Details>\\n\\n<Details>\\n<Summary><strong>Tooling</strong></Summary>\\n\\n- [assertoor](https://github.com/ethpandaops/assertoor) - (more details about assertoor can be found in the [assertoor blog post](https://ethpandaops.io/posts/assertoor-introduction/))\\n- [tx_spammer](https://github.com/MariusVanDerWijden/tx-fuzz) - Basic transaction spamming tool which can help you generate random transactions to test transaction inclusion in the blocks\\n- [blob_spammer](https://github.com/MariusVanDerWijden/tx-fuzz) - Basic blob spamming tool which can help you generate random blobs to test the network\'s ability to handle large amounts of data\\n- [goomy_blob](https://github.com/ethpandaops/goomy-blob) - A more advanced blob spamming tool which can help you generate blobs with a specific structure to test the network\'s ability to handle structured data\\n- [el_forkmon](https://github.com/ethereum/nodemonitor) - A fork monitor tool which can help you monitor the execution layter clients for forks and reorgs\\n- [blockscout](https://github.com/blockscout/blockscout) - A block explorer tool which can help you explore the blocks and transactions on the network\\n- [beacon_metrics_gazer](https://github.com/dapplion/beacon-metrics-gazer) - A validator metrics tool which can help you monitor the performance of the validators on the network - requires `prometheus_grafana` to be running\\n- [dora](https://github.com/ethpandaops/dora) - A ligthweight beaconchain explorer\\n- [full_beaconchain_explorer](https://github.com/gobitfly/eth2-beaconchain-explorer) - A full beaconchain explorer, monitoring conensus and execution layer data\\n- prometheus_grafana - An additional tool that spins up a grafana with prometheus to help you monitor the performance of the devnet\\n- [blobscan](https://github.com/Blobscan) - A tool that can help you scan the network for blobs and analyze the data that is being sent around\\n- [dugtrio](https://github.com/ethpandaops/dugtrio) - A consensus layer load balancer\\n- [blutgang](https://github.com/rainshowerLabs/blutgang) - An execution layer load balancer\\n- [rpc-snooper](https://github.com/ethpandaops/rpc-snooper) - A Man-in-The-Middle tool that can help you snoop on the RPC calls that are being made between consensus and execution layer clients, as well as between the consensus layer client and the validator client\\n- [blobber](https://github.com/marioevz/blobber/tree/main) - A tool that helps you generate malicious blobs to test the network\'s ability to handle malicious data\\n- [ethereum-metrics-exporter](https://github.com/ethpandaops/ethereum-metrics-exporter) - A tool that helps you export the metrics from the ethereum clients to prometheus - requires `prometheus_grafana` to be running\\n- [xatu](https://github.com/ethpandaops/xatu) - A centralized Ethereum network monitoirng tool for data pipelining - more details about xatu can be found in the [xatu blog post](https://ethpandaops.io/posts/open-source-xatu-data/#what-is-xatu)\\n- mev - MEV workflow tooling is fully supported using `mev_params`.\\n</Details>"},{"id":"assertoor-introduction","metadata":{"permalink":"/posts/assertoor-introduction","source":"@site/blog/2024-04-18-assertoor-introduction/index.md","title":"Assertoor: Ethereum Testnet Testing Tool","description":"Assertoor is a versatile tool designed to facilitate testing on Ethereum networks by evaluating specific conditions to determine their success.","date":"2024-04-18T00:00:00.000Z","tags":[{"inline":true,"label":"testnet","permalink":"/posts/tags/testnet"},{"inline":true,"label":"tool","permalink":"/posts/tags/tool"},{"inline":true,"label":"sepolia","permalink":"/posts/tags/sepolia"},{"inline":true,"label":"holesky","permalink":"/posts/tags/holesky"},{"inline":true,"label":"kurtosis","permalink":"/posts/tags/kurtosis"}],"readingTime":12.22,"hasTruncateMarker":false,"authors":[{"name":"pk910","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/pk910","github":"https://github.com/pk910","twitter":"https://x.com/_pk910_","imageURL":"/img/team/pk910.jpeg","key":"pk910","page":null},{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null}],"frontMatter":{"slug":"assertoor-introduction","title":"Assertoor: Ethereum Testnet Testing Tool","description":"Assertoor is a versatile tool designed to facilitate testing on Ethereum networks by evaluating specific conditions to determine their success.","authors":["pk910","parithosh"],"tags":["testnet","tool","sepolia","holesky","kurtosis"],"image":"/img/blog/assertor-introduction.jpg","githubRepos":[{"name":"assertoor","url":"https://github.com/ethpandaops/assertoor"}],"relatedLinks":[{"name":"Assertoor Test Playbooks","url":"https://github.com/ethpandaops/assertoor-test/tree/master/assertoor-tests"},{"name":"Assertoor Wiki","url":"https://github.com/ethpandaops/assertoor/wiki#supported-tasks-in-assertoor"}]},"unlisted":false,"prevItem":{"title":"Kurtosis: A Deep Dive to Local Devnets","permalink":"/posts/kurtosis-deep-dive"},"nextItem":{"title":"Tracoor: Ethereum beacon data and execution trace explorer","permalink":"/posts/tracoor-debug-tool"}},"content":"## Introduction to Assertoor\\n\\n[Assertoor](https://github.com/ethpandaops/assertoor) is a versatile tool designed to facilitate testing on Ethereum networks by evaluating specific conditions to determine their success. Drawing its name from its primary function\u2014asserting conditions\u2014Assertoor simplifies everything from routine checks (e.g., whether all clients are proposing blocks and overall network health) to more intricate processes such as the handling of deposits, exits, and similar transactions.\\n\\nFor those familiar with protocol testing, think of Assertoor as an end-to-e
1nd, cross-client integration testing tool that operates at a higher abstraction level than Hive.\\n\\nAssertoor excels as a test orchestrator, leveraging a YAML file to manage a series of tests. Each test consists of a sequence of tasks executed in a specific order. Assertoor not only orchestrates these tests but also provides foundational tasks like \\"generate_deposit\\" built into the system. Users can customize the test playbook to perform particular actions or verify specific conditions, enhancing the flexibility and applicability of Assertoor in various testing scenarios.\\n\\n\\n## Key Components of Assertoor\\n\\nAssertoor combines the functionalities of monitoring and task orchestration into a single tool, adeptly crafted in Go. Let\'s go through the key components that assertoor provides:\\n\\n### Network Connection and Monitoring\\nAssertoor establishes connections to multiple Consensus and Execution Clients through their public HTTP RPC APIs, requiring no extra integration efforts from client teams. This method ensures Assertoor maintains a robust and comprehensive view of the network status. Designed to be resilient, it can adapt to individual client faults or unsupported RPC endpoints by switching to another client for the required information.\\n\\n### Task Execution and Test Playbooks\\nAssertoor excels in executing a variety of tasks in a predefined order, which are specified in a test playbook. These playbooks can orchestrate both parallel and sequential steps, incorporating dependencies much like a GitHub workflow but specifically tailored for Ethereum network interactions. The tool\'s capabilities range from simple shell scripts to intricate built-in logic that performs various network interactions and checks. For a comprehensive list of all available tasks and their usage, visit the [Assertoor Project Wiki](https://github.com/ethpandaops/assertoor/wiki#supported-tasks-in-assertoor).\\n\\n### Flexible Test Definitions\\nTests are defined and executed via YAML files, allowing users to specify tasks within the test configuration or sideload them from external URLs. This flexibility facilitates organized test management and is illustrated in our test playbooks repository at [Assertoor Test Playbooks](https://github.com/ethpandaops/assertoor-test/tree/master/assertoor-tests).\\n\\n### User Interface and API Integration\\nAssertoor also provides a user-friendly interface for real-time test management (start/cancel) and monitoring. The interface displays detailed test statuses and serves logs and configurations for all tasks defined in the test playbook, comparable to the GitHub workflow run details UI.\\n<img src=\\"/img/blog/assertoor_overview.jpg\\" />\\n\\nAdditionally, Assertoor offers programmatic access through an API to trigger tests, fetch test results, and logs, making it ideal for integration with other systems and CI pipelines.\\n\\n\\n## Utilizing Assertoor\\n\\nAssertoor is engineered as a standalone tool that seamlessly connects to externally managed nodes for executing tests on Ethereum testnets. A key application involves routinely running specified tests on public testnets to ensure consistent network behavior and stability.\\n\\nEg. One of these test is designed to thoroughly validate the entire validator lifecycle on holesky. This includes complex scenarios involving depositing, BLS key changes, exits, withdrawals, and slashings. The test performs 10 deposits and evaluates various scenarios like double deposits, BLS key changes both before and after activation or exit, as well as handling slashings and exits. This comprehensive testing is vital for ensuring that all aspects of the validator lifecycle are functioning correctly under different conditions. For those interested in the specifics of this testing approach, the test playbook is available on our GitHub repository at [Validator Lifecycle Test](https://github.com/ethpandaops/assertoor-test/blob/master/assertoor-tests/validator-lifecycle-test-v2.yaml).\\n\\n### Integration with Kurtosis\\n\\nTo enhance flexibility and reduce the need for extensive manual configuration, Assertoor is integrated into our [Ethereum Kurtosis package](https://github.com/kurtosis-tech/ethereum-package). Kurtosis, along with the Ethereum Kurtosis package, provides a robust and extremely flexible environment for quickly spinning up ephemeral local Ethereum networks. These networks comprise multiple Ethereum clients of various types and include several additional tools, such as explorers and, crucially, Assertoor.\\n\\nWithin the Kurtosis environment, Assertoor benefits from automatic configuration. The package handles most setup requirements, including providing all client RPC URLs, validator name ranges, and a pre-funded wallet. This integration allows developers to focus primarily on defining the test playbooks that Assertoor should execute.\\n\\n### Example Configuration and Execution\\n\\nBelow is an example of a Kurtosis args file (yaml) that configures a simple testnet with two client pairs and sets Assertoor to run two specific test playbooks:\\n\\n```yaml\\nparticipants:\\n - el_type: geth\\n cl_type: lighthouse\\n count: 1\\n - el_type: geth\\n cl_type: lodestar\\n count: 1\\nadditional_services:\\n - assertoor\\n - dora\\nassertoor_params:\\n run_stability_check: fal
1se\\n run_block_proposal_check: false\\n tests:\\n - https://raw.githubusercontent.com/ethpandaops/assertoor-test/master/assertoor-tests/block-proposal-check.yaml\\n - https://raw.githubusercontent.com/ethpandaops/assertoor-test/master/assertoor-tests/all-opcodes-test.yaml\\n```\\n\\nTo start the testnet, use the following command:\\n\\n`kurtosis run --enclave my-testnet github.com/kurtosis-tech/ethereum-package --args-file ./args.yaml --image-download always`\\n\\nOnce Kurtosis has finished setting up the testnet, it provides a concise summary of the started services, including a link to the Assertoor web UI & API:\\n\\n```\\n================================= User Services =================================\\nUUID Name Ports Status\\nf3a69b405cdb assertoor http: 8080/tcp -> http://127.0.0.1:32839 RUNNING\\n```\\n\\nThe provided URL can be used to access the Web UI and API, where users can monitor the live test progress, and start or cancel additional tests as needed.\\n<img src=\\"/img/blog/assertoor_test_details.jpg\\" />\\n\\n\\n## Understanding Test Playbooks in Assertoor\\n\\nAssertoor utilizes test playbooks written in YAML format to run a series of tests on Ethereum networks. These playbooks are pivotal for orchestrating the testing process, providing both flexibility and precision in defining test scenarios.\\n\\nWe have a wide range of these test playbooks in our [GitHub repository](https://github.com/ethpandaops/assertoor-test/tree/master/assertoor-tests), which can be used as examples and reference points to get started with writing your own test playbooks.\\n\\nLet\'s take a look at how these playbooks are structured and how they work:\\n\\n### Structure of Test Playbooks\\n\\nTest playbooks begin with a header that provides general information about the test, such as its ID, name, and any relevant configuration values. Here\'s what a typical playbook header looks like, including configuration options:\\n\\n```yaml\\nid: block-proposal-check\\nname: \\"Every client pair proposed a block\\"\\ntimeout: 20m\\nconfig:\\n validatorPairNames: []\\n```\\n\\nThe header sets foundational parameters for the test, such as its timeout period and default configuration settings. These values can be overridden at the time of test invocation, either manually via API or through global configurations in the Assertoor config. \\n\\nIn the example provided, the `validatorPairNames` setting is utilized, which, when run as part of the Ethereum package within Kurtosis, is automatically populated with the names of all client pairs that have validator keys. Additional values supplied by Kurtosis include `clientPairNames`, which lists all client pair names (regardless of validator keys), and `walletPrivkey`, which is the private key for a pre-funded wallet.\\n\\n### Defining Tasks in Playbooks\\n\\nFollowing the header, the playbook specifies a series of tasks. Each task corresponds to a built-in function within Assertoor, designed to perform specific actions or checks:\\n\\n```yaml\\ntasks:\\n- name: check_clients_are_healthy\\n title: \\"Check if at least one client is ready\\"\\n timeout: 5m\\n config:\\n minClientCount: 1\\n\\n- name: run_task_matrix\\n title: \\"Check block proposals from all client pairs\\"\\n configVars:\\n matrixValues: \\"validatorPairNames\\"\\n config:\\n runConcurrent: true\\n matrixVar: \\"validatorPairName\\"\\n task:\\n name: check_consensus_block_proposals\\n title: \\"Wait for block proposal from ${validatorPairName}\\"\\n configVars:\\n validatorNamePattern: \\"validatorPairName\\"\\n```\\n\\nEach task is clearly defined with a name, a title for clarity, and any necessary configurations, such as timeouts or specific conditions to check. Assertoor currently supports 28 distinct tasks, with the possibility of adding more as needed. A detailed list of available tasks can be found on the [Assertoor Wiki](https://github.com/ethpandaops/assertoor/wiki#supported-tasks-in-assertoor).\\n\\n### Nested Tasks and Execution Flows\\n\\nPlaybooks can nest tasks within one another to control the execution flow more granularly. This nesting allows tasks to be organized in various execution flows like running tasks in the background, concurrently, or as a matrix of tasks.\\n\\nIn the provided example, the `run_task_matrix` task is used to execute the child task `check_consensus_block_proposals` multiple times in parallel, each instance triggered by a unique validator pair name. This approach enables simultaneous monitoring of block proposals from different validators, enhancing the efficiency and scope of testing. The `run_task_matrix` acts as a parent task, coordinating and waiting for all child tasks to complete before proceeding, ensuring that all necessary checks are performed thoroughly and concurrently.\\n\\nThis structure not only makes the playbook highly adaptable but also ensures that complex network behaviors are tested comprehensively, providing a reliable and scalable solution for network testing.\\n\\n### Exploring Advanced Playbooks\\n\\nWhile the example above provides a basic overview of playbook capabilities, for a deeper insight into more complex scenarios, we encourage exploring the advanced playbooks available in our test playbook repository. These playbooks can be accessed here: [Assertoor Test Playbooks](https://github.com/ethpandaops/assertoor-test/tree/master/assertoor-tests).\\n\\nSome noteworthy playbooks include:\\n\\n- **[EOA Transactions Test](https://github.com/ethpandaops/assertoor-test/blob/master/assertoor-tests/eoa-transactions-test.yaml):** This playbook checks whether transactions can be sent through each connected client and whether they can be included by each validator. It is crucial for verifying the robustness of transaction processing across different clients and validators.\\n\\n- **[Validator Withdrawal Test](https://github.com/ethpandaops/assertoor-test/blob/master/assertoor-tests/validator-withdrawal-test.yaml):** This test verifies whether validators can change their withdrawal credentials. It ensures that each validator can propose blocks with BLS changes, and that these BLS changes can be sent through each client. Additionally, it checks that all clients can propose blocks with withdrawals, an essential aspect of validator lifecycle management.\\n\\n- **[Validator Slashing Test](https://github.com/ethpandaops/assertoor-test/blob/master/assertoor-tests/validator-slashing-test.yaml):** This playbook tests if various types of slashings can be processed by the network. It confirms that each validator can propose blocks containing these slashings and that the slashings can be submitted through each client, ensuring the network\'s ability to handle penalizations correctly.\\n\\nThese advanced playbooks provide a comprehensive testing framework to evaluate different functionalities and scenarios within Ethereum networks, ensuring thorough validation and performance assessment.\\n\\n\\
1n## Integrating Assertoor and Kurtosis into Build Pipelines\\n\\nTo further streamline the usage of Assertoor, Kurtosis, and the Kurtosis Ethereum package in build pipelines, we\'ve encapsulated all necessary setup and monitoring steps into two reusable GitHub Actions. These actions are designed to simplify the deployment and testing process, making it more accessible and efficient for continuous integration workflows.\\n\\n### Assertoor GitHub Action\\n\\nThe first action we offer is the [`assertoor-github-action`](https://github.com/ethpandaops/assertoor-github-action). This action is utilized to continuously poll the test execution status from a running Assertoor instance. It queries the API at regular intervals and waits for all tests to successfully complete. The action also provides detailed outputs regarding the test status and information about any failed tests, which can be used by subsequent steps in the workflow for further processing or notifications.\\n\\n### Kurtosis Assertoor GitHub Action\\n\\nOur second offering is the [`kurtosis-assertoor-github-action`](https://github.com/ethpandaops/kurtosis-assertoor-github-action). This comprehensive action handles the entire process, including setting up Kurtosis, starting the testnet, and monitoring the Assertoor status. Designed for ease of use yet retaining high flexibility, it supports various backends and can be customized to fit specific testing needs.\\n\\n#### Example Workflow\\n\\nSetting up a full-scale Ethereum testnet with multiple client pairs and executing Assertoor tests is made straightforward with the following GitHub Actions workflow:\\n\\n```yaml\\non:\\n workflow_dispatch:\\n\\nname: run-assertoor-test\\njobs:\\n assertoor:\\n runs-on: ubuntu-latest\\n steps:\\n - name: Checkout Repository\\n uses: actions/checkout@v4\\n - name: Kurtosis Assertoor GitHub Action\\n uses: ethpandaops/kurtosis-assertoor-github-action@v1\\n with:\\n ethereum_package_args: ./kurtosis-args.yaml\\n```\\n\\nThis example demonstrates how to integrate the Kurtosis Assertoor GitHub Action into a typical CI/CD pipeline, simplifying the process of deploying and testing Ethereum networks for developers. The only additional thing needed is a `kurtosis-args.yaml` file as described in the previous sections.\\n\\n\\n## Conclusion\\n\\nSince its inception in November 2023, Assertoor has quickly proven to be an indispensable tool in the Ethereum testing landscape. Despite its relatively new status, it has already made significant impacts by detecting numerous issues across various client implementations. This efficiency is particularly evident in our recent focus on testing compatibility between Validator Clients and Beacon Nodes. Assertoor has drastically simplified the process of testing multiple client combinations, which would otherwise have been a daunting and time-consuming task.\\n\\nBy leveraging Assertoor alongside our reusable GitHub Actions, testing all possible client pair combinations becomes as straightforward as initiating a single workflow run. You can see this process in action and its results, including expected errors from incompatible client pairs, in our [recent workflow execution](https://github.com/ethpandaops/assertoor-test/actions/runs/8712937458). The job logs and Assertoor\'s summaries provide clear insights into which client pairs are currently facing compatibility issues.\\n\\n### Real-World Use Cases\\n\\n- **Scheduled Tests with latest Client Releases**:\\nAssertoor is used for scheduled tests in the [Assertoor Test Repository](https://github.com/ethpandaops/assertoor-test). A workflow runs daily, spinning up various testnets with the latest release images of all possible execution layer & consensus layer client combinations. Assertoor tests these networks for basic functionality, including beacon chain interactions (BLS changes, Exits, withdrawals, slashings) and execution chain interactions (opcode & precompiles testing). These tests have identified critical bugs in client releases, underlining the tool\'s value.\\n\\n- **Scheduled Tests on Public Ethereum Testnets**:\\nAssertoor is used in a standalone instance for scheduled and recurring testing of public Ethereum testnets. For instance, we have a setup dedicated to the Holesky testnet, running through a full validator lifecycle test approximately every two weeks. This use case highlights Assertoor\'s capability to perform in-depth network functionality checks, contributing significantly to network stability and reliability.\\n\\n- **Use in CI Pipelines**:\\nAssertoor, in combination with Kurtosis, is integrated into the CI pipeline of some Ethereum client development teams. This setup allows for the spinning up of testnets involving various client pairs to thoroughly test builds before release, ensuring compatibility and stability.\\neg. [rETH workflow integration](https://github.com/paradigmxyz/reth/blob/main/.github/workflows/assertoor.yml) / [Kurtosis Ethereum Package integration](https://github.com/kurtosis-tech/ethereum-package/blob/main/.github/workflows/per-pr.yml#L70-L78)\\n\\n\\nAs the Ethereum ecosystem continues to evolve, tools like Assertoor will be critical for ensuring that the network remains robust and secure. We look forward to further enhancing its features and expanding its use cases to meet the growing demands of blockchain technology."},{"id":"tracoor-debug-tool","metadata":{"permalink":"/posts/tracoor-debug-tool","source":"@site/blog/2024-04-12-tracoor-debug-tool/index.md","title":"Tracoor: Ethereum beacon data and execution trace explorer","description":"Tracoor is a tool that captures, stores and makes available critical data about the Ethereum network. This data can be invaluable when issues arise.","date":"2024-04-12T00:00:00.000Z","tags":[{"inline":true,"label":"debug","permalink":"/posts/tags/debug"},{"inline":true,"label":"tool","permalink":"/posts/tags/tool"},{"inline":true,"label":"tracoor","permalink":"/posts/tags/tracoor"},{"inline":true,"label":"mainnet","permalink":"/posts/tags/mainnet"},{"inline":true,"label":"sepolia","permalink":"/posts/tags/sepolia"},{"inline":true,"label":"holeksy","permalink":"/posts/tags/holeksy"}],"readingTime":3.79,"hasTruncateMarker":false,"authors":[{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","t
1witter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"tracoor-debug-tool","title":"Tracoor: Ethereum beacon data and execution trace explorer","description":"Tracoor is a tool that captures, stores and makes available critical data about the Ethereum network. This data can be invaluable when issues arise.","authors":["savid","samcm"],"tags":["debug","tool","tracoor","mainnet","sepolia","holeksy"],"image":"/img/blog/tracoor-debug-tool.jpg","githubRepos":[{"name":"tracoor","url":"https://github.com/ethpandaops/tracoor"}],"relatedLinks":[{"name":"Tracoor Mainnet","url":"https://tracoor.mainnet.ethpandaops.io"},{"name":"Tracoor Sepolia","url":"https://tracoor.mainnet.ethpandaops.io"},{"name":"Tracoor Hoodi","url":"https://tracoor.hoodi.ethpandaops.io"}]},"unlisted":false,"prevItem":{"title":"Assertoor: Ethereum Testnet Testing Tool","permalink":"/posts/assertoor-introduction"},"nextItem":{"title":"Open Sourcing Xatu Data","permalink":"/posts/open-source-xatu-data"}},"content":"import Alert from \'@site/src/components/Alert\';\\n\\n[**Tracoor**](https://tracoor.mainnet.ethpandaops.io) is a tool that **captures**, **stores** and **makes available** critical data about the Ethereum network, including:\\n\\n- [**Beacon States**](https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Debug/getStateV2)\\n- [**Beacon Blocks**](https://ethereum.github.io/beacon-APIs/?urls.primaryName=dev#/Beacon/getBlockV2)\\n- [**Execution Debug Traces**](https://geth.ethereum.org/docs/interacting-with-geth/rpc/ns-debug#debugtraceblock)\\n- [**Execution Bad Blocks**](https://geth.ethereum.org/docs/interacting-with-geth/rpc/ns-debug#debuggetbadblocks)\\n- Invalid gossiped verified **Blocks** and **Blobs**\\n\\nThis data can be invaluable when issues arise, allowing developers, researchers and hobbyists to quickly identify and address network problems.\\n\\n<img alt=\\"Tracoor homepage\\" class=\\"rounded\\" src=\\"/img/blog/homepage.jpg\\" />\\n\\nCheck out our **live** versions:\\n\\n- [**Mainnet**](https://tracoor.mainnet.ethpandaops.io)\\n- [**Sepolia**](https://tracoor.sepolia.ethpandaops.io)\\n- [**Holesky**](https://tracoor.holesky.ethpandaops.io)\\n\\nWe run the following clients:\\n\\n- Consensus layer: [nimbus](https://github.com/status-im/nimbus-eth2), [lighthouse](https://github.com/sigp/lighthouse), [prysm](https://github.com/prysmaticlabs/prysm), [lodestar](https://github.com/ChainSafe/lodestar) and [teku](https://github.com/Consensys/teku)\\n- Execution layer: [geth](https://github.com/ethereum/go-ethereum), [erigon](https://github.com/ledgerwatch/erigon), [besu](https://github.com/besu-eth/besu), [nethermind](https://github.com/NethermindEth/nethermind) and [reth](https://github.com/paradigmxyz/reth)\\n\\nWe store the data on mainnet for ~3 days to save storage costs and less for the testnets. `beacon bad blocks`, `beacon bad blobs` and `execution bad blocks` are stored for ~2 weeks.\\n\\n\\n<Alert>\\nWe strongly encourage anyone to run Tracoor and capture data for themselves and the wider community. Check out our [GitHub repo](https://github.com/ethpandaops/tracoor) for instructions.\\n</Alert>\\n\\n\\n## Tool helpers\\n\\nWe also provide an easy way to help consume our data with open source tools.\\n\\nIf you have any tools that could be used with the data we capture, please let us know in our [GitHub](https://github.com/ethpandaops/tracoor/issues) or [Twitter](https://twitter.com/ethpandaops).\\n\\n### GoEVMLab tracediff\\n\\n[GoEVMLab tracediff](https://github.com/holiman/goevmlab/tree/master/cmd/tracediff) allows you to load evm traces and find differences between transactions. [Tracoor](https://tracoor.mainnet.ethpandaops.io/go_evm_lab_diff) helper allows you select two blocks that were captured from different execution layer clients, select the transaction index in the block and then generate the commands required to diff the two transactions with the [GoEVMLab tracediff](https://github.com/holiman/goevmlab/tree/master/cmd/tracediff) tool.\\n\\n<img alt=\\"GoEVMLab tracediffe\\" class=\\"rounded\\" src=\\"/img/blog/goevmlab.jpg\\" />
1\\n\\n### lcli state transition\\n\\n[lcli state transition](https://github.com/sigp/lighthouse/tree/stable/lcli) allows for replaying state transitions from SSZ files to assist in fault-finding. [Tracoor](https://tracoor.mainnet.ethpandaops.io/lcli_state_transition) helper allows you select a beacon state and beacon block and then generate the commands required to replay the state transition with the [lcli state transition](https://github.com/sigp/lighthouse/tree/stable/lcli) tool.\\n\\n<img alt=\\"lcli state transition\\" class=\\"rounded\\" src=\\"/img/blog/lcli.jpg\\" />\\n\\n### ncli state transition\\n\\n[ncli state transition](https://github.com/status-im/nimbus-eth2/tree/stable/ncli) allows you to perform state transition given a pre-state and a block to apply. [Tracoor](https://tracoor.mainnet.ethpandaops.io/ncli_state_transition) helper allows you select a beacon state and beacon block and then generate the commands required to perform the state transition with the [ncli state transition](https://github.com/status-im/nimbus-eth2/tree/stable/ncli) tool.\\n\\n<img alt=\\"ncli state transition\\" class=\\"rounded\\" src=\\"/img/blog/ncli.jpg\\" />\\n\\n### zcli state diff\\n\\n[zcli state diff](https://github.com/protolambda/zcli) allows you to diff spec data, especially beacon states. [Tracoor](https://tracoor.mainnet.ethpandaops.io/zcli_state_diff) helper builds off [lcli state transition](#lcli-state-transition) or [ncli state transition](#ncli-state-transition) helpers by taking the state transition of those tools and comparing it to another beacon state we have captured with the [zcli state diff](https://github.com/protolambda/zcli) tool.\\n\\n<img alt=\\"zcli state diff\\" class=\\"rounded\\" src=\\"/img/blog/zcli.jpg\\" />\\n\\n## Conclusion\\n\\nCapturing and storing key Ethereum network data is vital for a healthy ecosystem. [**Tracoor**](https://github.com/ethpandaops/tracoor) provides quick access to beacon states, blocks, execution traces and more - empowering our community to quickly identify issues and develop solutions. \\n\\nWhile we hope to never need it, this proactive approach allows for swift and effective incident response.\\n\\nLove,\\n\\nEthPandaOps Team \u2764\ufe0f\\n\\n<Alert>\\nWhile we normally don\'t host any mainnet data that isn\'t for research purposes, we\'re doing so in this case to help debug the network in an emergency.\\n\\nIdeally if a healthy ecosystem develops similar to [checkpointz](https://eth-clients.github.io/checkpoint-sync-endpoints/), we\'d like to hand over hosting responsibility to the community.\\n\\nIf you can, please consider hosting your own instance of [Tracoor](https://github.com/ethpandaops/tracoor). If you\'re interested and need help, reach out to us on [Twitter](https://twitter.com/ethpandaops).\\n</Alert>"},{"id":"open-source-xatu-data","metadata":{"permalink":"/posts/open-source-xatu-data","source":"@site/blog/2024-03-25-open-source-xatu-data/index.md","title":"Open Sourcing Xatu Data","description":"We\'re thrilled to share that the EthPandaOps Xatu dataset is now open source! The dataset contains a wealth of information about the Ethereum network, including detailed data on beacon chain events, mempool activity, and canonical chain events.","date":"2024-03-25T00:00:00.000Z","tags":[{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"},{"inline":true,"label":"data","permalink":"/posts/tags/data"},{"inline":true,"label":"mainnet","permalink":"/posts/tags/mainnet"},{"inline":true,"label":"sepolia","permalink":"/posts/tags/sepolia"},{"inline":true,"label":"holesky","permalink":"/posts/tags/holesky"}],"readingTime":5.15,"hasTruncateMarker":false,"authors":[{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null}],"frontMatter":{"slug":"open-source-xatu-data","title":"Open Sourcing Xatu Data","description":"We\'re thrilled to share that the EthPandaOps Xatu dataset is now open source! The dataset contains a wealth of information about the Ethereum network, including detailed data on beacon chain events, mempool activity, and canonical chain events.","authors":["savid","samcm"],"tags":["xatu","data","mainnet","sepolia","holesky"],"image":"/img/blog/open-source-xatu-data.jpg","githubRepos":[{"name":"xatu-data","url":"https://github.com/ethpandaops/xatu-data"}],"relatedLinks":[{"name":"Apache Parquet","url":"https://parquet.apache.org"},{"name":"Clickhouse","url":"https://clickhouse.com/"}]},"unlisted":false,"prevItem":{"title":"Tracoor: Ethereum beacon data and execution trace explorer","permalink":"/posts/tracoor-debug-tool"},"nextItem":{"title":"Attacknet: Chaos engineering on Ethereum","permalink":"/posts/attacknet-introduction"}},"content":"import Alert from \'@site/src/components/Alert\';\\nimport { Details, Summary } from \'@site/src/components/Collapsible\';\\n\\n<Alert>\\nThe Ethereum Foundation is running a Data Challenge for EIP4844! [Click here](https://esp.ethereum.foundation/data-challenge-4844) for more info.\\n</Alert>\\n\\n## Introduction\\n\\nWe\'re thrilled to share that the EthPandaOps Xatu dataset is now open source!\\n\\nThe dataset contains a wealth of information about the Ethereum network, including detailed data on beacon chain events, mempool activity, and canonical chain events.\\n\\nSummary:\\n\\n- The data is dedicated to the public domain under the CC BY 4.0 license\\n- The entire schema is available [here](/data/xatu#schema)\\n- Data is partitioned by hour or day in [Apache Parquet](https://parquet.apache.org) files\\n- We\'ve already published the last 60 days of data for Mainnet, Holesky & Sepolia\\n - We\'ll publish everything we have for Mainnet over the following weeks -- it\'s just a lot of data!\\n - Testnets will have the last 90 days of data published\\n - Check [here](/data/xatu) for detailed data availability\\n- We\'ll continue to update tables with new data, as well as add new tables\\n\\n## Why are we open sourcing the data?\\n\\nWe aim to empower researchers, developers, and enthusiasts to explore the Ethereum network in depth and contribute to its ongoing evolution. We believe that open access to high-quality data is essential for the success of the Ethereum ecosystem.\\n\\nBy making this data publicly available we hope to foster greater understanding of the Ethereum network and drive advancements in areas such as protocol development, monitoring and performance optimization.\\n\\n## What is Xatu?\\n\\n[Xatu](https://github.com/ethpandaops/xatu) is a tool for collecting data from different c
1omponents of the Ethereum network that we\'ve been building for a while now. Since it\'s initial release in December 2022 we\'ve been running Xatu internally to monitor the Ethereum network by storing data in [Clickhouse](https://clickhouse.com/).\\n\\nIn-house we\'re using it for monitoring, analysis and incident response. Notably it was used for the [Big Blocks Test](https://notes.ethereum.org/@samcm/big-blocks) on Goerli/Mainnet in 2023 to help decide the EIP4844 blob parameters. It has also been the first port of call for analysing how Dencun performed through the fork lifecycle of [Devnets](https://notes.ethereum.org/@ethpandaops/dencun-devnet-10-analysis) -> [Testnets](https://notes.ethereum.org/@ethpandaops/goerli-sf-0-analysis) -> [Mainnet](../dencun-fork-analysis).\\n\\n## What\'s in the dataset?\\n\\nXatu has it\'s fingers in a lot of pies so we categorize the data into a few different types. Check [here](/data/xatu#schema) for the complete schema and data availability date ranges.\\n\\n### Beacon API Events\\n\\n<Alert>\\nLast 60 days published\\n</Alert>\\n\\nEvents that are derived from the [Beacon API Event Stream](https://ethereum.github.io/beacon-APIs/#/Events/eventstream) via Xatu Sentry from all consensus clients in multiple regions and networks. All events are annotated with additional data to help with analysis. For example, the `attestation` events have information about *when* the attestation was seen, and even the `validator_index` of the attestation. **Mainnet data exists from June 2023.**\\n\\n- 650+ billion `attestation` events\\n - 6TiB compressed, 300TiB uncompressed \ud83d\ude32\\n- 50+ million `block` events\\n- 50+ million `blob_sidecar` events\\n- Plus more!\\n\\n### Mempool Events\\n\\n<Alert>\\nPublishing soon <sup>tm</sup>\\n</Alert>\\n\\nEvents that are derived from Xatu Mimicry which connects to the execution p2p network. We\'ll be publishing these events in the next few days. **Mainnet data exists from March 2023.**\\n\\n- 3+ billion `transaction` events\\n\\n### Canonical Events\\n\\n<Alert>\\nPublishing soon <sup>tm</sup>\\n</Alert>\\n\\nWe also derive events from the finalized chain which we call `canonical` events. **Mainnet data exists from Beacon Chain genesis in December 2020.**\\n\\nThese events are especially useful for analysis when compared to `Beacon API Events` and `Mempool Events`. For example, comparing when an attestation was seen on the network against when it was included in a beacon block, or comparing when a transaction was first seen in the mempool to when it was included in a block. We\'ll be publishing these events in the coming weeks.\\n\\n\\n## How do I use the data?\\n\\nThe data is stored in Apache Parquet files which are a columnar storage format that is highly optimized for analytics. You can read these files using a variety of tools including Python & Clickhouse. Check out the [repo](https://github.com/ethpandaops/xatu-data) for more information on how to get started.\\n\\n### Clickhouse\\n\\nUsing Clickhouse is the simplest way to get started. You can use the `clickhouse client` to query the data directly from the Parquet files. Check out the [Clickhouse documentation](https://clickhouse.com/docs/en/install) to get setup.\\n\\n#### Querying directly\\nTo query all attestations events in Sepolia on the 20th of March 2024 in the 13th hour you can use the following query:\\n<Details>\\n<Summary>Query</Summary>\\n```sql\\nSELECT COUNT(*)\\nFROM\\n url(\'https://data.ethpandaops.io/xatu/sepolia/databases/default/beacon_api_eth_v1_events_attestation/2024/2/20/13.parquet\', \'Parquet\')\\n```\\n</Details>\\n\\n<img alt=\\"Querying Parquet directly from Clickhouse\\" class=\\"rounded\\" src=\\"/img/blog/example-1.jpg\\" />\\n\\n#### Inserting data\\nYou can also insert the data into a Clickhouse database to query it more easily. This is highly recommended for larger queries.\\n<Details>\\n<Summary>Query</Summary>\\n```sql\\nINSERT INTO default.beacon_api_eth_v1_events_attestation\\nSELECT *\\nFROM url(\'https://data.ethpandaops.io/xatu/sepolia/databases/default/beacon_api_eth_v1_events_attestation/2024/2/20/13.parquet\', \'Parquet\')\\n```\\n</Details>\\n\\n<img alt=\\"Inserting directly in to Clickhouse\\" class=\\"rounded\\" src=\\"/img/blog/example-3.jpg\\" />
1\\n\\n#### Globbing\\nClickhouse supports globbing so you can query multiple Parquet files at once. For example, to count the entire days worth of attestation events in Sepolia on the 20th of March 2024 you can use the following query:\\n<Details>\\n<Summary>Query</Summary>\\n```sql\\nSELECT COUNT(*)\\nFROM\\n url(\'https://data.ethpandaops.io/xatu/sepolia/databases/default/beacon_api_eth_v1_events_attestation/2024/2/20/{0..23}.parquet\', \'Parquet\')\\n```\\n</Details>\\n\\n<img alt=\\"Querying Parquet directly from Clickhouse with globbing\\" class=\\"rounded\\" src=\\"/img/blog/example-2.jpg\\" />\\n\\n## Conclusion\\n\\nWe cannot wait to see what the community does with the data! If you cook something up please let us know! We\'d love to hear about it. If you have any questions feel free to reach out to us on [Twitter](https://twitter.com/ethpandaops). If you notice any issues please make an issue on the [repo](https://github.com/ethpandaops/xatu-data/issues).\\n\\nHappy querying and don\'t forget the [Data Challenge](https://esp.ethereum.foundation/data-challenge-4844) run by the Ethereum Foundation!\\n\\nLove,\\n\\nEthPandaOps Team \u2764\ufe0f"},{"id":"attacknet-introduction","metadata":{"permalink":"/posts/attacknet-introduction","source":"@site/blog/2024-03-18-attacknet-introduction/index.md","title":"Attacknet: Chaos engineering on Ethereum","description":"Chaos testing is a disciplined approach to testing a system by proactively simulating and identifying failures. Ethereum networks in the wild are subject to a lot of real life variances that have historically been difficult to capture in local tests.","date":"2024-03-18T00:00:00.000Z","tags":[{"inline":true,"label":"mainnet","permalink":"/posts/tags/mainnet"},{"inline":true,"label":"tool","permalink":"/posts/tags/tool"},{"inline":true,"label":"deneb","permalink":"/posts/tags/deneb"},{"inline":true,"label":"dencun","permalink":"/posts/tags/dencun"},{"inline":true,"label":"cancun","permalink":"/posts/tags/cancun"},{"inline":true,"label":"chaos","permalink":"/posts/tags/chaos"}],"readingTime":3.41,"hasTruncateMarker":false,"authors":[{"name":"pk910","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/pk910","github":"https://github.com/pk910","twitter":"https://x.com/_pk910_","imageURL":"/img/team/pk910.jpeg","key":"pk910","page":null},{"name":"barnabasbusa","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/barnabasbusa","github":"https://github.com/barnabasbusa","twitter":"https://x.com/BarnabasBusa","imageURL":"/img/team/barnabasbusa.jpeg","key":"barnabasbusa","page":null},{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null}],"frontMatter":{"slug":"attacknet-introduction","title":"Attacknet: Chaos engineering on Ethereum","description":"Chaos testing is a disciplined approach to testing a system by proactively simulating and identifying failures. Ethereum networks in the wild are subject to a lot of real life variances that have historically been difficult to capture in local tests.","authors":["pk910","barnabasbusa","parithosh"],"tags":["mainnet","tool","deneb","dencun","cancun","chaos"],"image":"/img/blog/attacknet-introduction.jpg","githubRepos":[{"name":"ethereum-package","url":"https://github.com/ethpandaops/ethereum-package"},{"name":"crytic/attacknet","url":"https://github.com/crytic/attacknet"}],"relatedLinks":[{"name":"Trail of Bits","url":"https://www.trailofbits.com/"},{"name":"Chaos Mesh","url":"https://chaos-mesh.org/"}]},"unlisted":false,"prevItem":{"title":"Open Sourcing Xatu Data","permalink":"/posts/open-source-xatu-data"},"nextItem":{"title":"Dencun Fork Analysis","permalink":"/posts/dencun-fork-analysis"}},"content":"## Introduction\\nChaos testing is a disciplined approach to testing a system by proactively simulating and identifying failures. Ethereum networks in the wild are subject to a lot of real life variances that have historically been difficult to capture in local tests. We\'ve worked with [Trail of Bits](https://www.trailofbits.com/) to create a tool that c
1ombines the two worlds together, allowing us to create local networks that can simulate real world chaos and failures. Examples would include adding network latency between nodes, killing nodes at random, filesystem errors being returned, etc\\n\\nSuch a tool has two major components: \\n 1. A tool to deploy Ethereum networks \\n 2. A tool to orchestrate the chaos tests. \\n\\nWe have already solved 1. with [Kurtosis](https://github.com/kurtosis-tech/ethereum-package), it allows us to reliably deploy Ethereum networks (irrespective of which fork we\'re testing) to perform local docker based tests or remote Kubernetes based tests. To solve 2., we used a kubernetes testing framework called [Chaos Mesh](https://chaos-mesh.org/) which allows us to configure a host of failures that are then executed by the orchestrator. While these pieces of software give us the functionality we need, they don\'t provide the ease of use.\\n\\nIntroducing [Attacknet](https://github.com/crytic/attacknet), the missing piece was a software that can ingest a test definition and execute it for us. This would allow us to configure lists of cases we would like to see and have the execution of the test taken care of us. The software can also perform healthchecks and attempt to ascertain if a network was negatively affected by the chaos or not. \\n\\n<img src=\\"/img/blog/attacknet_diagram.jpg\\" />\\n\\n## Chaos types\\n\\n[Attacknet](https://github.com/crytic/attacknet) allows us to configure the following types of chaos for any duration we like and some also allow additional randomness (i.e, run fault only x% of time):\\n- Network: \\n * Network latency\\n * Jitter \\n * Packet drop\\n * Packet corruption\\n * Bandwidth limits\\n- Network split and re-joining\\n- Clock skews (i.e, NTP faults)\\n- CPU load\\n- RAM Load\\n- Input/Output: \\n * Disk faults \\n * Disk access latency\\n * I/O mistakes \\n- Service crashes (i.e, kills)\\n- WIP: Kernel faults \\n\\n## Test Scenarios\\nCurrently there are 3 factors we are playing with, i.e: Client combo in the network, Type of Chaos and Parameters of chaos. Here are some initial tests we planned:\\n\\n| Test Number | Description | Chaos Type | Parameters | Analysis Objective | Notes |\\n|-------------|------------------------------------|-------------------------|----------------------------------------------------------|--------------------------------------------------------------|-------------------------------------------------------------------------------------------|\\n| 1 | Single EL type, One of each CL. | Network latency | Latency: 100ms, 500ms, 1000ms, 2000ms with 10% jitter | Analyze the number of missed slots and correlate with latency | Repeat with every EL |\\n| 2 | Single EL type, One of each CL. | Network packet drop | Packet Drop: 5%, 10%, 50%, 100% | Analyze the number of missed slots and correlate with packet drop | Repeat with every EL |\\n| 3 | Single EL type, One of each CL. | Clock skew | Clock Skew: 100ms, 500ms, 1000ms, 2000ms | Analyze the number of missed slots and correlate with clock skew | Repeat with every EL (EL shouldn\u2019t matter in this test) |\\n| 4 | Single EL type, One of each CL. Apply partition to one CL at a time. | Network partition | N/A | Analyze if the CL can rejoin the network after the split is healed | Potentially use static peers to avoid downscoring issues. Restart of pods might be necessary. Repeat for every EL. |\\n| 5 | Single EL type, One of each CL. Split network into two halves. | Network partition | N/A | Analyze if the network can self heal | Use static peers to avoid downscoring issues. Repeat for every EL. |\\n\\n\\
1nWe\'re trying to be vary of adding too many variables into the mix, we don\'t have good ways of knowing when a run \\"fails\\", so we\'re erring on the side of cation and building networks with more controlled changesets.\\n\\n## Conclusion\\nAttacknet allowed us to simulate a lot of edge case test scenarios for the Dencun fork. We\'re going to continue using the tool for research into peerDAS as well as for hardening Ethereum clients. The next focus of the team will be to setup automated testing with attacknet, to allow us to run it without manual oversight. \\n\\nFind more information about Attacknet here: https://github.com/crytic/attacknet"},{"id":"dencun-fork-analysis","metadata":{"permalink":"/posts/dencun-fork-analysis","source":"@site/blog/2024-03-15-dencun-dork-analysis/index.md","title":"Dencun Fork Analysis","description":"Dencun has finally shipped to mainnet! This post will give some initial insights on how the fork event unfolded, and how it\'s performing.","date":"2024-03-15T00:00:00.000Z","tags":[{"inline":true,"label":"mainnet","permalink":"/posts/tags/mainnet"},{"inline":true,"label":"deneb","permalink":"/posts/tags/deneb"},{"inline":true,"label":"dencun","permalink":"/posts/tags/dencun"},{"inline":true,"label":"cancun","permalink":"/posts/tags/cancun"},{"inline":true,"label":"xatu","permalink":"/posts/tags/xatu"}],"readingTime":7.66,"hasTruncateMarker":false,"authors":[{"name":"parithosh","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/parithosh","github":"https://github.com/parithosh","twitter":"https://x.com/parithosh_j","website":"https://parithosh.com/","imageURL":"/img/team/parithosh.jpeg","key":"parithosh","page":null},{"name":"samcm","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/samcm","github":"https://github.com/samcm","twitter":"https://x.com/samcmAU","imageURL":"/img/team/samcm.jpeg","key":"samcm","page":null},{"name":"savid \ud83d\ude82","title":"DevOps Engineer","bio":"DevOps Engineer","url":"https://github.com/savid","github":"https://github.com/savid","twitter":"https://x.com/savid","imageURL":"/img/team/savid.jpeg","key":"savid","page":null}],"frontMatter":{"slug":"dencun-fork-analysis","title":"Dencun Fork Analysis","description":"Dencun has finally shipped to mainnet! This post will give some initial insights on how the fork event unfolded, and how it\'s performing.","authors":["parithosh","samcm","savid"],"tags":["mainnet","deneb","dencun","cancun","xatu"],"image":"/img/blog/dencun-fork-analysis.jpg","githubRepos":[{"name":"xatu","url":"https://github.com/ethpandaops/xatu"}]},"unlisted":false,"prevItem":{"title":"Attacknet: Chaos engineering on Ethereum","permalink":"/posts/attacknet-introduction"}},"content":"import Alert from \'@site/src/components/Alert\';\\n\\n<Alert>\\nThe Ethereum Foundation is running a Data Challenge for EIP4844! If you\'d like access to the data used in this analysis send an email to ethpandaops(at)ethereum.org\\n\\nMore info: https://esp.ethereum.foundation/data-challenge-4844\\n</Alert>\\n\\n## Introduction\\nDencun has finally shipped to mainnet! This post will give some initial insights on how the fork event unfolded, and how it\'s performing.\\n \\nThis analysis will cover roughly ~1d before the Dencun fork and ~1d after the fork. \\n\\n### Parameters\\n- Network: Ethereum Mainnet\\n- Fork: Dencun \\n- Load: Free range, all organic blob and transaction load \ud83c\udf43\\n \\n## Data sources\\nData was collected from 2 different sets of nodes. Each analysis section will note which data source used.\\n\\n### Xatu\\n\\nData was collected with the use of our distributed monitoring tool called [Xatu](https://notes.ethereum.org/@ethpandaops/xatu-overview). Xatu has a low resource use component called a Sentry that can collect data from the consensus client event stream and forward it to the Xatu server for collection. \\n\\nWe hosted 45 such sentries across 6 locations in the world, including: Santa Clara, San Francisco, Amsterdam, Helsinki, Bengaluru and Sydney. The consensus client side consisted of 9x each production ready consensus layer client. \\n\\nData from Xatu is available for research purposes. Send an email to `ethpandaops(at)ethereum.org` if you\'d like access!\\n\\n### Self-Hosted NUCs\\nWe run a large part of our mainnet stack in house on NUCs, this is akin to hardware that most solo stakers possess. This however should be taken with a grain of salt as the resource might vary depending on quite a lot of smaller factors. NUCs run from Europe and the \ud83e\udd85USA\ud83e\udd85.\\n\\n## Analysis\\n### Resource Usage\\n\\n> Data Source: Self-Hosted NUCs\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/cpu.jpg\\" />
1\\n\\n> [Link to CPU graph](https://grafana.observability.ethpandaops.io/d/MRfYwus7k/nodes?orgId=1&from=1710250242000&to=1710423042000&var-consensus_client=All&var-execution_client=All&var-network=mainnet&var-filter=instance%7C%21%3D%7Cmainnet-prysm-nethermind-001&var-filter=instance%7C%21%3D%7Cmainnet-lighthouse-geth-001&var-filter=instance%7C%21~%7C.%2Abootnode.%2A&var-filter=instance%7C%21%3D%7Cgballet-staking-1&viewPanel=2)\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/ram.jpg\\" />\\n\\n> [Link to RAM graph](https://grafana.observability.ethpandaops.io/d/MRfYwus7k/nodes?orgId=1&from=1710250242000&to=1710423042000&var-consensus_client=All&var-execution_client=All&var-network=mainnet&var-filter=instance%7C%21%3D%7Cmainnet-prysm-nethermind-001&var-filter=instance%7C%21%3D%7Cmainnet-lighthouse-geth-001&var-filter=instance%7C%21~%7C.%2Abootnode.%2A&var-filter=instance%7C%21%3D%7Cgballet-staking-1&viewPanel=9)\\n\\nCPU and RAM usage seems mostly unchanged across the clients over. This is likely due to the blob load being quite small throughout this period and the extensive optimisations made by the client teams \ud83d\udd25\\n\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/network-use.jpg\\" />\\n\\n> [Link to network graph](https://grafana.observability.ethpandaops.io/d/MRfYwus7k/nodes?orgId=1&from=1710329155885&to=1710347786717&var-consensus_client=All&var-execution_client=All&var-network=mainnet&var-filter=instance%7C%21%3D%7Cmainnet-prysm-nethermind-001&var-filter=instance%7C%21%3D%7Cmainnet-lighthouse-geth-001&var-filter=instance%7C%21~%7C.%2Abootnode.%2A&var-filter=instance%7C%21%3D%7Cgballet-staking-1&viewPanel=17)\\n\\nNetwork usage is also largely unchanged for these nodes, but will likely slightly increase as blob usage increases.\\n\\n\\n### Beacon Chain\\n\\n> Data Source: Self-Hosted NUCs + Xatu\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/network-participation.jpg\\" />\\n\\n> [Link to network participation](https://grafana.observability.ethpandaops.io/d/MRfYwus7k/nodes?orgId=1&from=1710250242000&to=1710423042000&var-consensus_client=All&var-execution_client=All&var-network=mainnet&var-filter=instance%7C%21%3D%7Cmainnet-prysm-nethermind-001&var-filter=instance%7C%21%3D%7Cmainnet-lighthouse-geth-001&var-filter=instance%7C%21~%7C.%2Abootnode.%2A&var-filter=instance%7C%21%3D%7Cgballet-staking-1&viewPanel=36)\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/offline-attesters.jpg\\" />\\n\\nWe see a 5% drop in participation immediately after the fork. Using a combination of [Blockprint](https://github.com/sigp/blockprint) and co-relating it with Xatu sentry data we are able to determine that there was an even spread of offline attesters across consensus layer clients. This indicates that there was no specific issue with a cleint but instead a general \\"forgot to update\\" situation.\\n\\n\\nIn terms of network health the drop reduced participation but quickly picked up and we are currently seeing similar levels as pre-fork.\\n\\n### Blobs\\n> Data Source: Xatu\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/blob-overall.jpg\\" />\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/blob-distr.jpg\\" />\\n\\n> [Link to blob distribution graph](https://grafana.observability.ethpandaops.io/d/be15cc
156-e151-4268-8772-f4a9c6a4e246/blobs?orgId=1&from=1710253039599&to=1710425839599&viewPanel=345)\\n> \\nBased on the blob metrics per block, we mostly see 0 blob blocks being built. This is likely due to most L2s not having their governance proposals to start using blobs. We can expect the blob usage to increase as more L2s start posting blobs. \\n\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/blob-arrival.jpg\\" />\\n\\n> [Link to blob arrival graph](https://grafana.observability.ethpandaops.io/d/be15cc56-e151-4268-8772-f4a9c6a4e246/blobs?orgId=1&from=1710253089666&to=1710425889666&viewPanel=184)\\n\\nThe small number of blobs on the network lead to them being on average propagated across all the sentries in < 2.5s, with the p95 case being ~4s. **This is close to the attestation boundary, one should probably keep a close eye on this information.**\\n\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/blob-heatmap.jpg\\" />\\n\\n> [Link to blob heatmap](https://grafana.observability.ethpandaops.io/d/be15cc56-e151-4268-8772-f4a9c6a4e246/blobs?orgId=1&from=1710253186535&to=1710425986535&viewPanel=185)\\n\\nThe blob heatmap mirrors the above point as well. Once there is more regular blob usage, we should be able to narrow down latency impacts on proposals.\\n\\n\\nBoth the above graphs indicate that we might need to narrow down if there is a certain region with far worse latency (like Australia).\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/blob-diffs.jpg\\" />\\n\\n> [Link to blob time difference graph](https://grafana.observability.ethpandaops.io/d/be15cc
156-e151-4268-8772-f4a9c6a4e246/blobs?orgId=1&from=1710253703079&to=1710426503079&viewPanel=231)\\n\\nThe time difference from the first blob seen to the last blob seen in a slot is quite narrow. This is good news as a higher latency would indicate some routing issues on the p2p layer. \\n\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/blob-per-slot.jpg\\" />\\n\\n> [Link to blob propagation per slot](https://grafana.observability.ethpandaops.io/d/be15cc56-e151-4268-8772-f4a9c6a4e246/blobs?orgId=1&from=1710253759997&to=1710426559997&viewPanel=195)\\n\\nThe numbers indicate at this load level that slots including 6 blobs or 1 blob doesn\'t have any noticeable impact on the average arrival time. \\n\\n<img class=\\"rounded\\" src=\\"/img/blog/blob-fill-rate.jpg\\" />\\n\\n> [Link to blob fill percentage](https://grafana.observability.ethpandaops.io/d/be15cc56-e151-4268-8772-f4a9c6a4e246/blobs?orgId=1&from=1710253759997&to=1710426559997&viewPanel=627)\\n\\nMajority of blobs contain little or no empty space.\\n\\n#### Blob impact on Blocks\\n> Data Source: Xatu\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/block-per-blob.jpg\\" />\\n\\n> [Link to block propagation vs number of blobs](https://grafana.observability.ethpandaops.io/d/be15cc
156-e151-4268-8772-f4a9c6a4e246/blobs?orgId=1&from=1710253726629&to=1710426526629&viewPanel=217)\\n\\nBlock arrival time seems to be is ~10% worse when a slot has 6 blobs compared to 1 blob. This is still well under the 4s mark, but we are clearly getting closer to the attestation deadline and more analysis might be needed.\\n\\n### Attestations\\n> Data Source: Xatu\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/attestation-heatmap.jpg\\" />\\n\\n> [Link to attestation heatmap](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&from=1710253872932&to=1710426672932&viewPanel=75)\\n> \\nThe attestation heatmap seems largely unchanged before and after the fork. Note the bright spot can be ignored - this is just a quirk of the data pipeline tool which ingested some duplicates through that period.\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/attestation-arrival.jpg\\" />\\n\\nThere is an increase in the floor of `min` arrival time as attesters wait for any blobs referenced in the block to arrive before creating their attestation.\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/attestation-arrival-p50.jpg\\" />\\n\\nThe 50th percentile of attestation arrivals remains the same as before the fork.\\n\\n### Blocks\\n> Data Source: Xatu\\n\\n<img class=\\"rounded\\" src=\\"/img/blog/block-arrival.jpg\\" />\\n\\n> [Link to block arrival graph](https://grafana.observability.ethpandaops.io/d/MR_U64N4k/beacon-api-events?orgId=1&from=1710250242000&to=1710423042000&viewPanel=36)\\n\\nThere is no major change in block arrival times after the fork. This isn\'t particularly surprising since the blob parameters are fairly conservative, but the indirect effects of more bandwidth usage were a concern. This may indicate that we can safely bump the target blob count in a future fork, but will need further analysis over a larger time period.\\n\\n\\n## Conclusion\\nUneventful forks are the best - a huge shoutout to all involved. This result comes off the back of 2+ years of R&D, and paints an extremely bright outlook for the Ethereum community.\\n\\nThings we expected:\\n- \u2705 Blobs arrive within an acceptable time\\n- \u2705 Slight delay in attestation arrivals due to the addition of blobs\\n- \u2705 No massively noticable increase in block arrival times\\n\\nSome questions will be answered as Dencun matures over time:\\n- \u2753 Is there a higher reorg rate from the increased bandwidth\\n- \u2753 Is there an impact to attestation effectiveness\\n - \u2753 Does it impact geographically distributed validators more\\n- \u2753 Can we safely bump the blob parameters\\n\\n**Wen Pectra?**"}]}}')}}]);
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.