On this Veterans Day, we’d like to extend our gratitude to all of those who have served in the U.S. military. Thank you for your service and sacrifices.
StreamSets is lucky to have Veterans on our team, and we’ve noticed some important overlap in skills for success in the military with those needed for success in data engineering.
What is a Data Engineer?
As data plays a larger role in business, the number of data scientists and analysts continue to grow. Behind every dashboard, every data-driven decision, and every data-based experience is a data pipeline, built by—you guessed it—a data engineer.
The data engineer is the technical professional who understands how data analysts and data scientists need data, then builds the data pipeline(s) to deliver the right data, in the right format, to the right place. The best data engineers are able to anticipate the needs of the business, track the rise of new technologies, and maintain a complex and evolving data infrastructure.
Demand for data engineers is up 50% according to a 2020 report from Datanami, making it one of the fastest growing jobs in the country. LinkedIn lists 66,000+ open data engineer jobs in the United States, and Indeed.com has over 80,000 with an average base salary of $131K.
So whether you’re a military Veteran exploring your next career or a recruiter or manager hiring for one of these hot jobs, read on to find out why Veterans make great data engineers.
When it comes to data today, there’s one constant: change. From ever evolving technology, to data drift, to changed data sources or endpoints, you will run into changing situations on a daily basis. For example, a change in IP address format disrupts data to a BI dashboard or a transition from 10-digit to 12-digit IDs affects 18,000 known applications. If you’re not prepared, these things can go unnoticed for months, with major ramifications. But when you know to assume and prepare for these inevitable changes, it can be as simple as getting an alert and porting your pipeline rather than rewriting it.
This is familiar territory for Veterans. “No plan survives contact with the enemy,” is a quote attributed to many great military leaders, in various forms. Military members may have personal experience being dropped off course, given broken supplies or some other type of unplanned event, that if change wasn’t assumed it would have had disastrous consequences. But since change is planned for as a core tenant of training and operations, it becomes less disruptive to the mission and the business of forcing change on the enemy. By knowing there will be change, knowing their roles, and being prepared, the military makes it work. Much like the most successful data engineers and data scientists.
In the realm of data engineering, a relatively new profession, the learning never ends. Modern data platforms are rapidly evolving, and there are a lot of them. Datanami lists these as the top 5 technical skills needed for data engineers: Python, SQL, big data, Hadoop, and ETL. But this is a fast changing ecosystem, and Scala and machine learning are the next in-demand skills.
In fact, the average lifespan of a data platform today is only a few years. You may also need to understand a variety of infrastructures that work with data platforms–on-premises, hybrid, and cloud–and need to learn APIs and new business logic regularly. Data engineering is always changing and the ability to learn quickly will serve you well.
Picking up a skill quickly is nothing new to a Veteran. “In a 6-month period, you are asked to master something that most people will take 4 years to get a degree in,” said Spencer Milo, a former sergeant in the US Army. “The fact that you can learn whatever is put in front of you is a really important skill.”
Skilled in Risk Assessment
Starting in boot camp, risk assessment is a high priority. Active military can find themselves in highly volatile situations, responsible for personnel and materiel. Learning to look out for themselves, while considering their team, is a must. Recruits go through many classes and training exercises on quickly assessing risk, so it becomes second nature to them out in the field.
This is a highly valued skill for data engineers, who must constantly think about the consequences of what they are doing and how it may put other workloads at risk. Failed pipelines mean corrupted data, missing data, or broken applications. Analysts or business partners could be making business decisions based on bad data (and not ever know it) or you may be losing customers due to broken applications. Finding efficient ways to de-risk data pipelines is part of a successful data engineer’s toolkit.
Data engineers are force multipliers that enhance the fidelity and accuracy of business analytics. Much like members of the military they need to be highly collaborative team players, who want to help others on their team succeed. Their role balances data management with analysis, and must work with data scientists closely. Good data engineers don’t get stuck in the build- break-fix cycle so common to traditional data pipelines and ETL. They find a way to enable their teammates with self-service solutions, so that as 1 data engineer, they can support 10s of ETL developers who can support 100s of data analysts.
This structure will sound familiar to Veterans; the military is a team-based organization, through and through. From the first day in bootcamp, through the final mission, military personnel cooperate for safety and to solve problems and meet goals. And the military is nothing without its efficiency and brother/sisterhood.
Data engineers have to find their own path to meet the goals of their stakeholders. No one hands them a roadmap to the solution. Creativity, grit, and perseverance allow data engineers to be effective problem-solvers for the organization. It’s important they easily grasp the problem and find a solution. Just as important, data engineers need to find a way to reduce time spent designing and deploying data pipelines (so they can work on those business problems). The more resilient your data pipelines, the more time you have to work on the big problems that matter most.
Military members are familiar with being handed a mission and resources (often too few) in order to meet that mission. Whether leader or soldier, there’s no room for complaining about problems. If you see a challenge that needs a solution, you step up and get it done.
Take the Leap
It should be pretty clear at this point that a) data engineering can be a very cool, challenging, and rewarding career (especially with the right tools); and b) Veterans have the right stuff for the job. I’d encourage you to explore the match further. So, Vets—search LinkedIn jobs for those postings. Recruiters and hiring managers, actively look for Vets or move them up your list. You’ll be thrilled with the results!