Numerous organizations are receiving Hadoop in their IT framework. For old Big Data stagers with a solid building group, it is typically not a major issue to plan the objective framework, pick an innovation stack, and start implementation. Those with a ton of experience can in any case in some cases face obstructions with all the intricacy, however Hadoop novices face a bunch of difficulties to begin. The following are the most usually observed Hadoop challenges which Grid Dynamics settle for its customers.
Diversity of Vendors. Which to choose?
The common first reaction is to use the original Hadoop binaries from the Apache website but this results in the realization as to why only a few companies use them “as-is” in a production environment. There are a lot of great arguments to not do this. But then panic comes with the realization of just how many Hadoop distributions are freely available from Hortonworks, Cloudera, MapR and ending with large commercial IBM InfoSphere BigInsights and Oracle Big Data Appliance. Oracle even includes hardware! Things become even more tangled after a few introductory calls with the vendors. Selecting the right distribution is not an easy task, even for experienced staff, since each of them embed different Hadoop components (like Cloudera Impala in CDH), configuration managers (Ambari, Cloudera Manager, etc.), and an overall vision of a Hadoop mission.
SQL on Hadoop. Very popular, but not clear…
Hadoop stores a lot of data. Apart from processing according to predefined pipelines, businesses want to get more value by giving an interactive access to data scientists and business analysts. Marketing buzz on the Internet even forces them to do this, implying, but not clearly saying, competitiveness with Enterprise Data Warehouses. The situation here is similar to the diversity of vendors, since there are too many frameworks that provide “interactive SQL on top of Hadoop,” but the challenge is not in selecting the best one. Understand that currently they all are still not an equal replacement for traditional OLAP databases. Simultaneously with many obvious strategic advantages, there are disputable shortcomings in performance, SQL-compliance, and support simplicity. This is a different world and you should either play by its rules or do not consider it as a replacement for traditional approaches.
Big Data Engineers. Are there any?
A good engineering staff is a major part of any IT organization, but it is really critical in Big Data. Relying on good Java/Python/C++/etc engineers to design/implement good quality data processing flows in most of cases means wasting of millions of dollars. After two years of development you could get unstable, unsupportable, and over-engineered chaotic scripts/jars accompanied by a zoo of frameworks. The situation becomes desperate if key developers leave the company. As in any other programming area, experienced Big Data developers spend most of the time thinking how to keep things simple and how the system will evaluate in the future. But experience in the Big Data technological stack is a key factor. So the challenge is in finding such developers.
Secured Hadoop Environment. Point of a headache.
More and more companies are storing sensitive data in Hadoop. Hopefully not credit cards numbers, but at least data which falls under security regulations with respective requirements. So this challenge is purely technical, but often causes issues. Things are simple if there are only HDFS and MapReduce used. Both data-in-the-motion and at-rest encryption are available, file system permissions are enough for authorization, Kerberos is used for authentication. Just add perimeter and host level security with explicit edge nodes and be calm. But once you decide to use other frameworks, especially if they execute requests under their own system user, you’re diving into troubles. The first is that not all of them support Kerberized environment. The second is that they might not have their own authorization features. The third is frequent absence of data-in-the-motion encryption. And, finally, lots of trouble if requests are supposed to be submitted outside of the cluster.
Conclusion
We brought up a couple of effective difficulties as we see them. Obviously, the things above are a long way from being finished and one could be frightened away by them bringing about a choice to not utilize Hadoop at all or to delay its appropriation for some later time. That would not be shrewd. There is an entire rundown of advantages brought by Hadoop to associations with handy hands. In participation with other Big Data systems and procedures, it can move capacities of the information arranged business to a completely new degree of execution.
Source:https://dzone.com/articles/what-are-the-biggest-hadoop-challenges