Zeppelin at twitter (sf data science meetup, july 2016)
•Download as PPTX, PDF•
4 likes•1,112 views
Zeppelin at Twitter (SF Data Science Meetup, July 2016)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to Zeppelin at twitter (sf data science meetup, july 2016)
Similar to Zeppelin at twitter (sf data science meetup, july 2016) (20)
Recently uploaded
Recently uploaded (20)
Zeppelin at twitter (sf data science meetup, july 2016)
- 1. Zeppelin at Twitter Prasad Wagle Technical Lead, Data Platform twitter.com/prasadwagle July 13, 2016 SF Data Science Meetup Galvanize, San Francisco, CA
- 2. Twitter Data Pipeline Overview Production systems Presto Vertica MySQL Scalding Spark R Custom Dashboards Tableau Zeppelin Command line tools HDFS Analytics Tools Analytics Front-ends
- 3. One company-wide server 860 notes 4000 paragraphs 1500 Vertica, 1500 Presto, 300 MySQL 400 Markdown 300 Scalding, Spark, etc 850 users Zeppelin Usage Metrics
- 4. Field of (Data) Dreams Started as hackweek project in Dec 2015, Beta: Jan 27 Number of notes created Feb - 270, Mar - 350, Apr - 470, May - 560, Jun - 750
- 5. Report Creators Product managers (dashboards, product analytics) Data scientists Sales analysts Engineers and SREs Report Viewers Anyone in the company Zeppelin Users
- 6. My mind is absolutely blown away by the ease of use, speed, and power of Zeppelin. I've been wanting a tool like this at Twitter my entire time working here. started playing with @ApacheZeppelin. amazingly addictive! Thanks for all the updates to Zeppelin - Fabric has fallen in love with it fast (and we're even using it for daily tracking of our OKRs amongst all the other metrics) Zeppelin Testimonials
- 7. Very easy to create and share reports Web based Works seamlessly with analytics engines JDBC Non-JDBC - Scalding, Spark Open source (easy to add features) Reasons for adoption
- 8. Drag and drop report builder Can create complex queries without SQL knowledge (e.g. Top N) Polished UI Filters and other transformations work on extracts no new database queries (fast) Row level permissions (for sales reports) Tableau
- 10. Authentication Integrated with Twitter’s homegrown single sign-on system SSL Integrated with Twitter’s homegrown key distribution system Notebook authorization Data source authorization Work Done (Security)
- 11. Websocket deadlock issue with Jetty 8 reduce communication remove synchronized block (risky, will move to Jetty 9) Monitoring Standby server Backups Work Done (Stability, Operations)
- 12. Scalding JDBC Create JDBC connection for every query to avoid Vertica closed connection issue Work Done (Interpreter)
- 13. Notebook authorization Data source authorization Run scheduled notes with a user Scalding interpreter Reduce websocket communication Paragraph footer Row level permissions Work Contributed to Apache Project
- 14. Stability, Scalability Jetty9 Interpreters Scalding, Spark, R Multiuser scalability, authentication, integration with Twitter sources Use JDBC interpreter instead of Hive Future / Work in Progress
- 15. Features and UX Notebook organization (folders) Email reports and alerts Row level permissions like tableau Operations Monitoring (end-to-end query) Admin (view/stop running jobs, resource usage) Failover Continuous Integration Future / Work in Progress
- 16. There is a real need for Notebook style interface for data analysis Zeppelin is enterprise ready, flexible and easy to use Zeppelin user and dev community is awesome Takeaways