Hi everyone, I’m Chris from Projecttent.
Projecttent has provided blockchain infrastructure services on a number of blockchain networks, on both testnets and mainnets since 2018. We started off running mining rigs in 2018 and added validator node services in 2019. While each network presents a different set of challenges, our approach has been to learn, document, and codify the experience gained on each network. A common theme in our approach to validation is redundancy, with respect to infrastructure, monitoring, and security.
As team principal, I have 15+ years of engineering experience as a software and hardware engineer working in early stage startups. The rest of the team consists of one DevOps engineer and one software developer.
Experience
We run validator nodes on a number of PoS chains, including Tendermint chains:
- Celo
- Nucypher
- Keep
- Near
- Solana
- Mina
- Avalanche
- Band Protocol
- Certik
- Microtick
- Althea
We run 30+ dedicated servers in multiple regions. Our cloud infrastructure is geographically distributed across multiple datacenters for redundancy and added reliability. Our nodes maintain a 99.9% uptime performance on the various networks we support.
Security
We use a secure remote signing server paired with internal security protocols, which we enforce in software, to protect validator keys and prevent double signing on Tendermint networks. This practice ensures that validator nodes have no access to sensitive private keys. As we continue to grow our service on Tendermint networks, we will move to rely on an HSM key storage solution to further reduce our attack surface.
We run our Tendermint validators behind multiple sentry nodes. Both validator and sentry nodes include protection against denial of service attacks. In cases where a server must respond to HTTP requests (e.g. Celo attestation servers), we add an extra layer of defense with Cloudflare DDoS protection. Where applicable, validator traffic is routed via proxy servers or sentry nodes (on Tendermint). Network traffic is filtered through both software and hardware firewalls, with special care taken to ensure those rules apply to Docker containers as well when we do run them.
All servers are hardened with a set of security practices, including restricted access requiring multi-factor authentication. We have undergone and passed security audits by a 3rd party.
Reliability
We run our custom Beacon monitoring software on all our nodes. It hooks into the various blockchain networks we support to monitor server health and validator performance. This service provides automated monitoring on all operational nodes. It sends alerts on CPU, storage, memory and network utilization, as well as on node specific metrics such as missed blocks, transactions, node status etc. On some networks, our Beacon software can query the blockchain for relevant transactions and combine those with validator-reported metrics for a more precise accounting of validator state. Beacon is configured to send alerts to our DevOps team, via PagerDuty and email. We also rely on Prometheus as a backup to our Beacon monitors.
Performance
We currently run validator nodes on 30+ bare metal single-tenant servers, across multiple datacenters. We perform routine chaindata backups and in some cases have replica or backup servers we can promote to the active validator set. Our servers are multi-core (8 or 12), with 64GB or 128GB memory capacity, and terabytes of storage. For networks where we don’t already have a running backup server, we have the capacity to provision new servers in an hour or less if required. We rely on Ansible and custom scripts to generate reproducible server builds that satisfy our validator node requirements.
Our validator nodes run, alongside our node monitoring software, on dedicated, bare metal servers. In all cases, we run on overprovisioned servers with spare compute, storage and memory capacity, to absorb spikes when they do occur. Our nodes are routinely among the highest performing nodes on the networks we run on, in excess of 99.9% uptime in all cases.
Governance
As infrastructure providers we’re often involved in the various development efforts that occur at the early stage of a network. These efforts range from active participation in network DAOs, to participation in and support of offshoot networks that sprout from a platform network, to contributing core or tooling software to the network.
Examples of community involvement include:
- Celo: genesis validator, grant recipient, active role in Celo governance, early community member in the various networks building on the Celo blockchain
- Arweave: genesis miner, founding member of the Arweave DAO; infrastructure provider on various networks building on Arweave
- Keep: grant recipient, tBTC software contributor, educational articles
- Microtick: IBC relayer (in progress)
Communication
Please feel free to reach out to me directly on Discord at chris | projecttent#6955 or via any of the following:
Thank you for your time. Please let me know if you have any questions.