The situation
Liv Up is a healthy meals startup based in Sao Paulo, Brazil. Using their website or app you can build a menu and have meals delivered straight to your door so you can prepare them at home easily and quickly. They started with frozen meals and are now expanding to other products such as juices, snacks and fully stocked fridges delivered to offices.
Liv Up was founded in 2016 and has rapidly grown to over 400 employees and a series C round of funding.
Liv Up is a very data driven company. They have a large operation and use data to control and operate the whole process; from sourcing raw materials, production, distribution, sales and customer support. Data is used at every stage to optimise processes. This means they can be agile and have a quick feedback loop on their product. They develop new products in less than a month and can get feedback to their kitchens in less than 15 days. However, it also means that there is a growing amount of complexity - there are lots of processes to control and data knowledge is becoming fragmented across the company.
The data team is responsible for empowering other decision makers in the business with data, analyses and recommendations. To have an impact on this decision making process the data team needed to overcome a few issues:
- Managing advanced workflows
- Scaling the data team
- Giving analysts autonomy
Managing advanced workflows
“Using multiple tools created a steep learning curve, particularly as new analysts didn’t always have a tech background. This slowed down learning.”
Before Liv Up started using Dataform they had a sophisticated system in place. They had already adopted SQL for transformations, they version controlled their code and they used Airflow to schedule their queries. However this process was happening across 3 or 4 different tools, which all had high technical bars. This meant when onboarding new members of the team, the learning curve was often very steep and it would take a long time to get a new team member up and running. Liv Up were going through a period of hypergrowth. They needed systems which were simple enough to be learned in days, not months.
“Dataform felt like it was designed exactly for our use case.”
With Dataform they can develop their code in SQL, write data quality tests, version control their code and schedule queries all inside one platform. They can also automatically manage dependencies (something that wasn’t happening in their previous setup). This saved Liv Up significant time onboarding new members of the team.
Liv Up wanted to be able to hire more domain driven people e.g. people who had experience in food product development. They are able to understand the context of the business better and therefore generate more insightful analyses compared to someone from a deeply technical background with no experience in the food industry. Using Dataform meant that analysts were no longer required to learn how to do things such as develop using the command line. Liv Up could now lower the technical bar of the people they were hiring and instead hire domain experts.
Scaling the data team
“One feature that is really important is the DAG - it means new people can onboard super easily.”
Liv Up were planning a period of intense growth, massively scaling the data team. They were going from about 6 analysts of 20. They wanted to make sure that as they grew, analysts could still easily find the data they were looking for and avoid data knowledge being held in one or two individuals heads. They needed a tool which meant that they could easily collaborate as a team and one which was easy to onboard new members.
“Dataform brings transparency about our data model. This really helps new people understand what’s going on.”
Dataform enabled Liv Up to centralize their teams data knowledge in a single place. Allowing them to document datasets and understand data lineage has made their data model much easier to navigate. The diagram linking all of the transformations (DAG) means that new team members can easily get up to speed with where all the tables live and how they relate to each other. Using Dataform’s Git integration, pull requests and code reviews help teams develop transformations together and keep up to date with other analysts changes. Liv Up have not only saved time onboarding new members but can now collaborate effectively within one platform.
Giving analysts autonomy
“The process is so well designed that we don’t really need anyone dedicated to overseeing or owning data engineering.”
Liv Up realised the importance of enabling their analysts to be autonomous. They wanted the data team to be able to make fast, well informed decisions and develop their own analyses. They realised if they didn’t find a solution to this then it would only lead to frustration and unnecessary process.
As the data team is a key enabler of decisions across the business, it was important that they could understand business context and develop new data models without requiring input from data engineering.
With Dataform, analysts own the entire data transformation process. Business analysts no longer need to ask a data engineer to adapt the data model for them as they can independently create and publish new tables and columns. These tables can then be re-used by other members of the team (and the data science team) saving time rewriting the same code. If there is an error in the pipeline then analysts have the full context and are easily able to find and fix it. Using Dataform’s assertions and built in version control meant that more senior members of the team could still review the code before it was pushed to check for errors.
Conclusion
Liv Up were going through a period of rapid growth and through using Dataform found a way to empower their analysts to be autonomous. They were able to scale the size of their team from 6 to 20 people by reducing the number of tools they were using and making their onboarding process more efficient. They saved money by not having to hire a team of dedicated data engineers to oversee the data transformation process and they can now hire analysts with strong domain expertise.
A future with Dataform...
“The main business outcome is that we are much faster to do things. We can develop data science models a lot faster than most other companies”
As a result of increased productivity and time saved, Liv Up are able to focus their time on larger initiatives like data science projects. They were developing a deep learning model that combines lots of features. Beforehand the data science team would have to ask multiple different people where to find the data. They now use the DAG in Dataform to look at where sessions, product impressions and sales information is. Using Dataform means that they can very easily speed up the process of finding the data needed to put into their models. Liv Up can develop data science models a lot faster than other companies.
Liv Up would also like to democratise Dataform’s access and make it more widely accessible to other teams in the company. They’d love to get marketing and engineering teams developing their own transformations and enabling more people in the company to make better decisions.
