hadoop - YARN: How to run MapReduce jobs with lot of mappers comparing to cluster size -

-

I have 1-node headop test setup with MapReduce job which starts 96 mapers and 6 reducers. Before migrating from the yarn, this job was stable but with normal yarn, it started to hang 100% with the majority of mappers in the 'pending' state.

The job is actually 6 Sub-Job (16 Mapper +1 rediscover each). This configuration shows the production process sequence. All of them are under a single job control, are there any such configurations that need to be examined, or are relatively big jobs compared to the nodes and cluster sizes of such small cases? The worst case for developers I can reduce the job with a group of sub-jobs, but I do not want to do this because there is no reason to do this on production and I must have the exam and production sequence same.

When I migrated to the YARN scheduler then it was changed to farsiibular and it is currently the only option as I run Claudera and Chloeida, strongly recommends that unbiased scheduler Therefore switching to the FIFO scheduler is not an option.

Any other option in addition to 'redefining the job'?

Disables current 'Qi per user' logic (switch to single queue) and assign the allocation file Having solved my problems with the limited amount of applications running running.

  • as yarn.scheduler.fair.user-as- The default-queue was set to false.
  • The dynamic resource allocation queue in Clodaria Manager was changed to 'Default' so that queue does not allow applications running more than 2. 1-node design is good enough for testing equipment. This allocation file will be improved in the open source.

So far it works as needed. Everything else except the default policy is left.


Comments

Post a Comment

Popular posts from this blog

import - Python ImportError: No module named wmi -

-
I followed the instructions for downloading WMI for Python When I try to run the code for C.Win32_Service (StartMode = "Auto", state = "Stopped") import wmi c = wmi.WMI (): if Raw_input ("restart% s?"% S.Caption) .upper () == "Y": s.StartService () I got the error < Code> traceback (most recent call end): file ". \", Line 1, edit: I am using Python 2.7.6 EDIT2: I am running 64-bit Windows 2008 R2, and I've downloaded WMI-1.4.9.zip (md5). I removed the content and put it in D: Python \ Tools \ Scripts I executed dragon setup.py.install I said D : \ Python \ tools \ scripts to% PATH% and when I execute the code for C.Win32_Service (StartMode = "Auto", state = "Stopped") import wmi c = Wmi.WMI (): If raw_input ("Restart% s?"% S. Caption). () == "Y": s.StartService () error me Gets traceback (most recent call final): File ". See \", line 1, &
Read more

Editing Python Class in Shell and SQLAlchemy -

-
I am working on the terminal on a shell script after this tutorial SQLAlchemy tutorial. I need to type gt; & Gt; & Gt; Sqlalchemy import column, integer, string & gt; & Gt; & Gt; Class user (base): __tablename__ = 'users' id = column (integer, primary_key = true) name = column (string) absolute name = column (string) password = column (string) def __repr __ (self): return "& lt ; User (name = '% s', absolute name = '% s', password = '% s') & gt; "% (Self.name, self.fullname, self.password) After the problem I have typed the password = column (string) I hit twice and changed .... >> I then took everything back but then an error was thrown because the class already exists ... I am not completely sure how to fix it. How do I open that slip in the shell script and edit it (Add DF to DRR) The error thrown below: / user / giripeters / TFSQLLX / Lib / python2.7 / site-packages / sqlalchemy / ext / decla
Read more

c# - MySQL Parameterized Select Query joining tables issue -

-
I am using parameterized select questions in conjunction with my program and it works entirely, When I try to join the table Everything I have done is a small fraction: using (MySqlCommand cmd = new MySqlCommand (paramQuery) ToSql (), Connection)) /etc/ paramQuery.ToSql () Equal: "Select TableOne ID, Tableon, Department, Tablet. One by name table, Table 2 h Yes TableOnEID = @Pharmak "* / for (Int Index = 0; Index and Lt; ParamariX Parameter (.c calculation; Index ++) Cmd.Parameters.AddWithValue (paramQuery.Parameters (). ElementAt (index). , ParamQuery.Parameters (). ElementAt (index) .Value); / * ParamQuery.Parameters (). ElementAt (index) .k = "@most 0" paramQuery.Parameters (). ElementAt (index) .Value = "tableTwo.ID" * / MySqlDataReader Reader = cmd.ExecuteReader (); While (reader.Read ()) {// do stuff} } One of the tables I attempt to join is For the table, two with the same id for all. Is there something that I'm doing wrong?
Read more