Facebook used it for its Inbox search. Other than Facebook, a number of other services use it, for example Digg. Facebook abandoned Cassandra back in but the solution has been used at Instagram since replacing Redis. Scribe was a flexible logging system that Facebook used for a multitude of purposes internally. Hadoop is an open source map-reduce implementation that makes it possible to perform calculations on massive amounts of data.
Facebook uses this for data analysis and as we all know, Facebook has massive amounts of data. Hive originated from within Facebook, and makes it possible to use SQL queries against Hadoop, making it easier for non-programmers to use. Both Hadoop and Hive are open source Apache projects and are used by a number of big services, for example Yahoo and Twitter. Facebook uses several different languages for its different services. Thrift is an internally developed cross-language framework that ties all of these different languages together, making it possible for them to talk to each other.
This has made it much easier for Facebook to keep up its cross-language development. Facebook has made Thrift open source and support for even more languages has been added. Varnish is an HTTP accelerator which can act as a load balancer and also cache content which can then be served lightning-fast. Facebook uses Varnish to serve photos and profile pictures, handling billions of requests every day.
Like almost everything Facebook uses, Varnish is open source. React is an open-source JavaScript library created in by Jordan Walke, a software engineer at Facebook. Later, Facebook introduced React Fiber, which is a collection of algorithms for rendering graphics. Read the story of how React became so successful. But handling such a large system is a complex task, so we thought we would list a few more things that Facebook does to keep its service running smoothly.
Facebook has a system they called Gatekeeper that lets them run different code for different sets of users it basically introduces different conditions in the code base.
This acts as a real-world stress test and helps expose bottlenecks and other problem areas before a feature is officially launched. This made it impossible for the rest of the internet to find our servers. All of this happened very fast. Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems.
But this took time, because these facilities are designed with high levels of physical and system security in mind. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online. Once our backbone network connectivity was restored across our data center regions, everything came back up with it.
But the problem was not over — we knew that flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic.
Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk.
In a storm exercise, we simulate a major system failure by taking a service, data center, or entire region offline, stress testing all the infrastructure and software involved.
We do not know what processors Facebook is using for Hadoop, but it seems likely that it is keeping close to parity between processor core count per node and the number of spindles attached to it. As you can see from the diagram, there is not a lot of empty space in this rack. The Leopard servers have a low 32 GB of memory per node, but each of the dozen nodes in the rack is allocated with an entire Knox Open Vault array, yielding 30 4 TB drives per node for TB total capacity.
We are amazed that Facebook has not moved to fatter disk drives for Haystack, but when you buy in bulk, you can probably stay off the top-end parts. This time last year, Jason Taylor, vice president of infrastructure foundation at Facebook, told The Next Platform that users were uploading 40 PB of photos per day at Facebook, and with its new push into streaming video, the rate of capacity expansion here must be enormous.
Obviously, with flash drives now pushing 10 TB, size is not the issue, but cost still is. Each Leopard server has GB of memory and has drives in total attached to it for a total of TB of capacity. The other drives in the rack are spun down and sitting quietly, waiting for an access. This allows Facebook to do 1 exabyte of storage in a power envelope of about 1.
Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between. Subscribe now. Public-private partnerships are common when responding to national or international crises and the current coronavirus pandemic that is expanding around the globe is no different. To account for this, Facebook has started a lot of open source projects and backend services.
Their philosophy for the creation of services is as follows:. A list of all of Facebook's open source developments can be found here. I will discuss a few of the essential tools that Facebook has developed.
Thrift is a lightweight remote procedure call framework for scalable cross-language services development.
It's quick, saves development time, and provides a division of labor of work on high-performance servers and applications. Scribe is a server for aggregating log data streamed in real-time from many other servers.
It is a scalable framework useful for logging a wide array of data. It is built on top of Thrift. Cassandra is a database management system designed to handle large amounts of data spread out across many servers. It powers Facebook's Inbox Search feature and provides a structured key-value store with eventual consistency.
0コメント