arundhajhttps://www.arundhaj.com/2022-11-06T11:00:00+05:30all that is technologyUsing Terraform, creating Thumbnail Image in AWS Lambda with AWS S3 trigger2022-11-06T11:00:00+05:302022-11-06T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2022-11-06:/blog/terraform-aws-lambda-s3-thumbnail-image.html<p>In this video, we will be writing an AWS Lambda function that creates a thumbnail image which is triggered whenever an image is uploaded to a S3 bucket with AWS S3 trigger and deployed using Terraform. <a href="https://github.com/CodePossibility/thumbnail_generation_lambda">Source code</a> can be found here.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/n8R2fg7CvDk" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>We'll first initialize the project</p>
<div class="highlight"><pre><span></span><code>$ <span class="nb">export</span> <span class="nv">AWS_PROFILE</span><span class="o">=</span>arundhaj …</code></pre></div><p>In this video, we will be writing an AWS Lambda function that creates a thumbnail image which is triggered whenever an image is uploaded to a S3 bucket with AWS S3 trigger and deployed using Terraform. <a href="https://github.com/CodePossibility/thumbnail_generation_lambda">Source code</a> can be found here.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/n8R2fg7CvDk" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>We'll first initialize the project</p>
<div class="highlight"><pre><span></span><code>$ <span class="nb">export</span> <span class="nv">AWS_PROFILE</span><span class="o">=</span>arundhaj
$ <span class="nb">cd</span> CodePossibility
$ mkdir thumbnail_generation_lambda <span class="o">&&</span> <span class="nb">cd</span> thumbnail_generation_lambda
</code></pre></div>
<p>Then start writing the lambda function <code>src\app.py</code>. This function upon triggered, open the original file in the source bucket, create thumbnail and save it in the destination bucket.</p>
<p>We can then start writing the terraform script in <code>main.tf</code>, <code>variables.tf</code> and <code>output.tf</code>, to create IAM Policies, Roles, Lambda, S3 bucket, S3 trigger and Cloudwatch.</p>
<p>Terraform commands used are:</p>
<div class="highlight"><pre><span></span><code>$ terraform init
$ terraform plan
$ terraform apply
$ terraform destory
</code></pre></div>
<p>AWS CLI commands used are:</p>
<div class="highlight"><pre><span></span><code>$ aws iam list-policies --query <span class="s1">'Policies[?PolicyName == `thumbnail_s3_policy`]'</span>
$ aws iam list-roles --query <span class="s1">'Roles[?RoleName == `thumbnail_lambda_role`]'</span>
$ aws lambda list-functions --query <span class="s1">'Functions[?FunctionName == `thumbnail_generation_lambda`]'</span>
$ aws s3 ls <span class="p">|</span> grep cp-
$ aws s3 ls s3://cp-original-image-bucket
$ aws s3 ls s3://cp-thumbnail-image-bucket
$ aws logs describe-log-groups --query <span class="s1">'logGroups[?logGroupName == `/aws/lambda/thumbnail_generation_lambda`]'</span>
$ aws logs tail /aws/lambda/thumbnail_generation_lambda --since 30s<span class="p">|</span>5m<span class="p">|</span>1h
$ aws s3 cp high_resolution_image.jpeg s3://cp-original-image-bucket
$ aws logs tail /aws/lambda/thumbnail_generation_lambda --since 30s<span class="p">|</span>5m<span class="p">|</span>1h
</code></pre></div>
<p>Hope this helps!</p>Routing in Python Flask and HTTP Methods2022-05-03T11:00:00+05:302022-05-03T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2022-05-03:/blog/routing-in-flask-http-methods.html<p>In this article and video, we’ll see how to write RESTful APIs using Python Flask for the required behavior. </p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/dl8UJnuGVjg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>The <code>route</code> decorator is used to define the behavior of APIs. It can be applied to a method or class. Below is the basic use of <code>route</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/user …</span></code></pre></div><p>In this article and video, we’ll see how to write RESTful APIs using Python Flask for the required behavior. </p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/dl8UJnuGVjg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>The <code>route</code> decorator is used to define the behavior of APIs. It can be applied to a method or class. Below is the basic use of <code>route</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/user'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">users</span><span class="p">():</span>
<span class="k">return</span> <span class="n">user_service</span><span class="o">.</span><span class="n">get_user_list</span><span class="p">()</span>
</code></pre></div>
<h3>Path Parameters</h3>
<p>Path params is passed in the URL of the request like <code>https://api.test-domain-name.com/project/b2713a81-e2da-429d-87ba-3d7a4a04975c</code> which will be handled by the following <code>route</code> as shown below</p>
<div class="highlight"><pre><span></span><code><span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'project/<uuid:id>'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">project</span><span class="p">(</span><span class="nb">id</span><span class="p">):</span>
<span class="k">return</span> <span class="n">project_service</span><span class="o">.</span><span class="n">get_project_detail</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span>
</code></pre></div>
<h3>Query Parameters</h3>
<p>Query params can be accessed with <code>request.args</code> variable which is parsed from the URL of format <code>https://api.test-domain-name.com/user?first_name=Arunkumar</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">request</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/user'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">users</span><span class="p">():</span>
<span class="n">user_filter</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="c1"># request.args itself is a dict, however, it is not a secure practice to use</span>
<span class="c1"># request data without cleaning or escaping or validating.</span>
<span class="k">if</span> <span class="s1">'first_name'</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="p">:</span>
<span class="n">user_filter</span><span class="p">[</span><span class="s1">'first_name'</span><span class="p">]</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'first_name'</span><span class="p">)</span>
<span class="k">if</span> <span class="s1">'last_name'</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="p">:</span>
<span class="n">user_filter</span><span class="p">[</span><span class="s1">'last_name'</span><span class="p">]</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'last_name'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">user_service</span><span class="o">.</span><span class="n">get_user_list</span><span class="p">(</span><span class="n">user_filter</span><span class="p">)</span>
</code></pre></div>
<h3>HTTP Methods</h3>
<p>Following are the most common HTTP methods, we may require to implement routes for:</p>
<ul>
<li>GET</li>
<li>POST</li>
<li>PUT</li>
<li>PATCH</li>
<li>DELETE</li>
<li>OPTIONS</li>
</ul>
<p>In Flask, requests method can be identified using <code>request.method</code> variable, as shown below</p>
<div class="highlight"><pre><span></span><code><span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/user'</span><span class="p">,</span> <span class="n">methods</span><span class="o">=</span><span class="p">[</span><span class="s1">'GET'</span><span class="p">,</span> <span class="s1">'POST'</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">users</span><span class="p">():</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s1">'GET'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">user_service</span><span class="o">.</span><span class="n">get_user_list</span><span class="p">()</span>
<span class="k">elif</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s1">'POST'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">user_service</span><span class="o">.</span><span class="n">add_user</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">'Unsupported HTTP Method'</span>
</code></pre></div>
<p>Class based routes can also be defined as below</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">flask.views</span> <span class="kn">import</span> <span class="n">MethodView</span>
<span class="k">class</span> <span class="nc">UsersAPI</span><span class="p">(</span><span class="n">MethodView</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">user_service</span><span class="o">.</span><span class="n">get_user_list</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">post</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">user_service</span><span class="o">.</span><span class="n">add_user</span><span class="p">()</span>
<span class="n">app</span><span class="o">.</span><span class="n">add_url_rule</span><span class="p">(</span><span class="s1">'/user'</span><span class="p">,</span> <span class="n">view_func</span><span class="o">=</span><span class="n">UserAPI</span><span class="o">.</span><span class="n">as_view</span><span class="p">(</span><span class="s1">'users'</span><span class="p">))</span>
</code></pre></div>
<h3>Requests and Response Headers</h3>
<p>Following snippet shows how to access Request headers with <code>request.headers</code> variable and set Response headers</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">request</span><span class="p">,</span> <span class="n">abort</span><span class="p">,</span> <span class="n">make_response</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/user'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">users</span><span class="p">():</span>
<span class="n">auth_token</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Authorization'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="k">if</span> <span class="n">auth_token</span> <span class="ow">or</span> <span class="n">auth_service</span><span class="o">.</span><span class="n">validate</span><span class="p">(</span><span class="n">auth_token</span><span class="p">):</span>
<span class="n">abort</span><span class="p">(</span><span class="mi">401</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">make_response</span><span class="p">(</span><span class="n">user_service</span><span class="o">.</span><span class="n">get_user_list</span><span class="p">())</span>
<span class="n">response</span><span class="o">.</span><span class="n">headers</span><span class="p">[</span><span class="s1">'Content-Type'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'application/json'</span>
<span class="k">return</span> <span class="n">response</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">after_request</span>
<span class="k">def</span> <span class="nf">after_request</span><span class="p">(</span><span class="n">response</span><span class="p">):</span>
<span class="n">response</span><span class="o">.</span><span class="n">headers</span><span class="p">[</span><span class="s1">'X-CUSTOM-API-HEADER'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'CustomHeaderValue'</span>
<span class="k">return</span> <span class="n">response</span>
</code></pre></div>
<p>Hope this helps!</p>Creating AWS S3 Presigned URL for uploading and downloading files in Python using Boto32022-02-10T18:00:00+05:302022-02-10T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2022-02-10:/blog/aws-presigned-url-python-boto3.html<p>In this video I'll show how create AWS S3 presigned URL for uploading and downloading files in Python using Boto3.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/MfDZKmL_PNo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>You can find the code snippet below</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
<span class="kn">import</span> <span class="nn">boto3</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">Session</span><span class="p">(</span><span class="n">profile_name</span><span class="o">=</span><span class="s1">'arundhaj'</span><span class="p">)</span>
<span class="n">s3_client</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
<span class="n">s3_bucket_name</span> <span class="o">=</span> <span class="s1">'codepossibility-presigned'</span>
<span class="n">object_name</span> <span class="o">=</span> <span class="s1">'TestUploadFile.txt'</span>
<span class="k">def</span> <span class="nf">create_bucket</span><span class="p">():</span>
<span class="n">response …</span></code></pre></div><p>In this video I'll show how create AWS S3 presigned URL for uploading and downloading files in Python using Boto3.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/MfDZKmL_PNo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>You can find the code snippet below</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
<span class="kn">import</span> <span class="nn">boto3</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">Session</span><span class="p">(</span><span class="n">profile_name</span><span class="o">=</span><span class="s1">'arundhaj'</span><span class="p">)</span>
<span class="n">s3_client</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
<span class="n">s3_bucket_name</span> <span class="o">=</span> <span class="s1">'codepossibility-presigned'</span>
<span class="n">object_name</span> <span class="o">=</span> <span class="s1">'TestUploadFile.txt'</span>
<span class="k">def</span> <span class="nf">create_bucket</span><span class="p">():</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">s3_client</span><span class="o">.</span><span class="n">create_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">s3_bucket_name</span><span class="p">)</span>
<span class="n">pprint</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">file_upload</span><span class="p">():</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">s3_client</span><span class="o">.</span><span class="n">generate_presigned_post</span><span class="p">(</span>
<span class="n">s3_bucket_name</span><span class="p">,</span>
<span class="n">object_name</span><span class="p">,</span>
<span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">3600</span>
<span class="p">)</span>
<span class="n">pprint</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">file_download</span><span class="p">():</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">s3_client</span><span class="o">.</span><span class="n">generate_presigned_url</span><span class="p">(</span>
<span class="s1">'get_object'</span><span class="p">,</span>
<span class="n">Params</span><span class="o">=</span><span class="p">{</span>
<span class="s1">'Bucket'</span><span class="p">:</span> <span class="n">s3_bucket_name</span><span class="p">,</span>
<span class="s1">'Key'</span><span class="p">:</span> <span class="n">object_name</span>
<span class="p">},</span>
<span class="n">ExpiresIn</span><span class="o">=</span><span class="mi">3600</span>
<span class="p">)</span>
<span class="n">pprint</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
<span class="c1"># create_bucket()</span>
<span class="c1"># file_upload()</span>
<span class="n">file_download</span><span class="p">()</span>
</code></pre></div>
<p>Hope this helps!</p>Using SQLAlchemy Expression for Insert2021-12-29T18:00:00+05:302021-12-29T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-12-29:/blog/sqlalchemy-expression-insert.html<p>In this article, I'll show how to build SQLAlchemy expressions for various cases of inserting data to the table</p>
<p>For simple insert,</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Prepare the statement</span>
<span class="n">insert_stmt</span> <span class="o">=</span> <span class="n">insert</span><span class="p">(</span><span class="n">TableName</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="p">(</span><span class="n">Column1</span><span class="o">=</span><span class="s1">'Column1Value'</span><span class="p">,</span> <span class="n">Column2</span><span class="o">=</span><span class="s1">'Column2Value'</span><span class="p">)</span>
<span class="c1"># Execute the statement</span>
<span class="n">insert_result</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">insert_stmt</span><span class="p">)</span>
<span class="c1"># Get the Primary Key of the newly inserted record …</span></code></pre></div><p>In this article, I'll show how to build SQLAlchemy expressions for various cases of inserting data to the table</p>
<p>For simple insert,</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Prepare the statement</span>
<span class="n">insert_stmt</span> <span class="o">=</span> <span class="n">insert</span><span class="p">(</span><span class="n">TableName</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="p">(</span><span class="n">Column1</span><span class="o">=</span><span class="s1">'Column1Value'</span><span class="p">,</span> <span class="n">Column2</span><span class="o">=</span><span class="s1">'Column2Value'</span><span class="p">)</span>
<span class="c1"># Execute the statement</span>
<span class="n">insert_result</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">insert_stmt</span><span class="p">)</span>
<span class="c1"># Get the Primary Key of the newly inserted record</span>
<span class="n">inserted_id</span> <span class="o">=</span> <span class="n">insert_result</span><span class="o">.</span><span class="n">inserted_primary_key</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</code></pre></div>
<p>At times, passing the values directly in <code>values</code> method would get complex whenever the columns we need use are dynamic. In those cases,</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Build the data dictionary</span>
<span class="n">insert_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="n">insert_data</span><span class="p">[</span><span class="s1">'Column1'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Column1Value'</span>
<span class="k">if</span> <span class="n">Column2</span><span class="p">:</span>
<span class="n">insert_data</span><span class="p">[</span><span class="s1">'Column2'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Column2Value'</span>
<span class="c1"># Prepare the statement</span>
<span class="n">insert_stmt</span> <span class="o">=</span> <span class="n">insert</span><span class="p">(</span><span class="n">TableName</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="p">(</span><span class="n">insert_data</span><span class="p">)</span>
<span class="c1"># Execute the statement</span>
<span class="n">insert_result</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">insert_stmt</span><span class="p">)</span>
<span class="c1"># Get the Primary Key of the newly inserted record</span>
<span class="n">inserted_id</span> <span class="o">=</span> <span class="n">insert_result</span><span class="o">.</span><span class="n">inserted_primary_key</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</code></pre></div>
<p>To insert bulk records in a single statement using a <code>bindparam</code>. In this case, all <code>bindparam</code> columns should be present and cannot be skipped. So, need to specify <code>None</code> explicitly.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Build the list of data dictionary</span>
<span class="n">insert_data_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="nb">input</span> <span class="ow">in</span> <span class="n">input_list</span><span class="p">:</span>
<span class="n">insert_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="n">insert_data</span><span class="p">[</span><span class="s1">'Column1'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">input</span><span class="p">[</span><span class="s1">'Column1'</span><span class="p">]</span>
<span class="n">insert_data</span><span class="p">[</span><span class="s1">'Column2'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">input</span><span class="p">[</span><span class="s1">'Column2'</span><span class="p">]</span> <span class="k">if</span> <span class="s1">'Column2'</span> <span class="ow">in</span> <span class="nb">input</span> <span class="k">else</span> <span class="kc">None</span>
<span class="n">insert_data_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">insert_data</span><span class="p">)</span>
<span class="c1"># Prepare the statement</span>
<span class="n">insert_stmt</span> <span class="o">=</span> <span class="n">insert</span><span class="p">(</span><span class="n">TableName</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="p">(</span>
<span class="n">Column1</span><span class="o">=</span><span class="n">bindparam</span><span class="p">(</span><span class="s1">'Column1'</span><span class="p">),</span> <span class="n">Column2</span><span class="o">=</span><span class="n">bindparam</span><span class="p">(</span><span class="s1">'Column2'</span><span class="p">))</span>
<span class="c1"># Execute the statement</span>
<span class="n">insert_result</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">insert_stmt</span><span class="p">,</span> <span class="n">insert_data_list</span><span class="p">)</span>
</code></pre></div>
<p>While inserting, if there is a duplicate record by Primary Key, we can <code>ignore</code> by specifying <code>prefix_with('IGNORE')</code></p>
<div class="highlight"><pre><span></span><code><span class="c1"># Build the data dictionary</span>
<span class="n">insert_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="n">insert_data</span><span class="p">[</span><span class="s1">'TableNameId'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">insert_data</span><span class="p">[</span><span class="s1">'Column1'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Column1Value'</span>
<span class="k">if</span> <span class="n">Column2</span><span class="p">:</span>
<span class="n">insert_data</span><span class="p">[</span><span class="s1">'Column2'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Column2Value'</span>
<span class="c1"># Prepare the statement</span>
<span class="n">insert_stmt</span> <span class="o">=</span> <span class="n">insert</span><span class="p">(</span><span class="n">TableName</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="p">(</span><span class="n">insert_data</span><span class="p">)</span><span class="o">.</span><span class="n">prefix_with</span><span class="p">(</span><span class="s1">'IGNORE'</span><span class="p">)</span>
<span class="c1"># Execute the statement</span>
<span class="n">insert_result</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">insert_stmt</span><span class="p">)</span>
<span class="c1"># Get the Primary Key of the newly inserted record, would be 0 if the insert was ignored</span>
<span class="n">inserted_id</span> <span class="o">=</span> <span class="n">insert_result</span><span class="o">.</span><span class="n">inserted_primary_key</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</code></pre></div>
<p>Hope this helps!</p>5 Responsive Layouts built with Angular FlexLayout2021-06-03T11:00:00+05:302021-06-03T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-06-03:/blog/5-responsive-layouts-built-angular-flexlayout.html<p>I came across an article <a href="https://www.freecodecamp.org/news/learn-flexbox-build-5-layouts">Learn CSS Flexbox by Building 5 Responsive Layouts</a> in which the author neatly explains the concept. As I'm extensively using <a href="https://angular.io">Angular</a>, <a href="https://material.angular.io">Angular Material</a> and <a href="https://github.com/angular/flex-layout">Angular FlexLayout</a>, I though of reproducing the layouts with FlexLayout.</p>
<h3 style="border-bottom: 0.18rem solid black;">Layout 1 - Responsive Card Layout</h3>
<p>This is a responsive card layout …</p><p>I came across an article <a href="https://www.freecodecamp.org/news/learn-flexbox-build-5-layouts">Learn CSS Flexbox by Building 5 Responsive Layouts</a> in which the author neatly explains the concept. As I'm extensively using <a href="https://angular.io">Angular</a>, <a href="https://material.angular.io">Angular Material</a> and <a href="https://github.com/angular/flex-layout">Angular FlexLayout</a>, I though of reproducing the layouts with FlexLayout.</p>
<h3 style="border-bottom: 0.18rem solid black;">Layout 1 - Responsive Card Layout</h3>
<p>This is a responsive card layout adjusting the number of cards based on the available width.</p>
<p><img alt="Layout 1 - Responsive Card Layout" src="/images/flexlayout-layout1.png"></p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="o">=</span><span class="s">"row wrap"</span> <span class="na">class</span><span class="o">=</span><span class="s">"container"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span> <span class="err">*</span><span class="na">ngFor</span><span class="o">=</span><span class="s">"let i of [1, 2, 3, 4, 5, 6, 7, 8, 9]"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span> <span class="na">fxLayout</span><span class="o">=</span><span class="s">"row"</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span><span class="p">></span>
{{ 'Item ' + i }}
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">container</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">.</span><span class="nc">grid-item</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="kc">lightsalmon</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">border-radius</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">min-width</span><span class="p">:</span><span class="w"> </span><span class="mi">300</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">height</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">margin</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<h3 style="border-bottom: 0.18rem solid black;">Layout 2 - Responsive Nav Bar Layout</h3>
<p>This is a responsive menu navigation bar that shows in <code>rows</code> for desktops and <code>columns</code> for mobiles.</p>
<p><img alt="Layout 2 - Responsive Nav Bar Layout" src="/images/flexlayout-layout2.png"></p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"row"</span> <span class="na">fxLayout</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"column"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>Home<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>About<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>Services<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>Contact<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">grid-item</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">.</span><span class="nc">grid-item</span><span class="p">:</span><span class="nd">hover</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="kc">lightgray</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">cursor</span><span class="p">:</span><span class="w"> </span><span class="kc">pointer</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<h3 style="border-bottom: 0.18rem solid black;">Layout 3 - Responsive Side Nav Layout</h3>
<p>This is a responsive side nav that adjusts left/right for desktops and top/bottom for mobiles.</p>
<p><img alt="Layout 3 - Responsive Side Nav Layout" src="/images/flexlayout-layout3.png"></p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"container"</span> <span class="na">fxLayout</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"row"</span> <span class="na">fxLayout</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"column"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"20%"</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"20%"</span>
<span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span> <span class="na">class</span><span class="o">=</span><span class="s">"left"</span><span class="p">></span>Left<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"80%"</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"80%"</span>
<span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span> <span class="na">class</span><span class="o">=</span><span class="s">"right"</span><span class="p">></span>Right<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">container</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">height</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">vh</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">.</span><span class="nc">left</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="kc">lightgray</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">.</span><span class="nc">right</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="kc">beige</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<h3 style="border-bottom: 0.18rem solid black;">Layout 4 - Responsive Card Layout2</h3>
<p>This is another format of responsive card layout.</p>
<p><img alt="Layout 4 - Responsive Card Layout2" src="/images/flexlayout-layout4.png"></p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="o">=</span><span class="s">"column"</span> <span class="na">fxLayoutGap</span><span class="o">=</span><span class="s">"10px"</span> <span class="na">class</span><span class="o">=</span><span class="s">"container"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="o">=</span><span class="s">"row"</span> <span class="na">fxFlex</span><span class="o">=</span><span class="s">"33%"</span> <span class="na">class</span><span class="o">=</span><span class="s">"block-1"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span> <span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>A<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"row"</span> <span class="na">fxLayout</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"column"</span>
<span class="na">fxLayoutGap</span><span class="o">=</span><span class="s">"10px"</span> <span class="na">fxFlex</span><span class="o">=</span><span class="s">"33%"</span> <span class="na">class</span><span class="o">=</span><span class="s">"block-2"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"70%"</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"50%"</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span>
<span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>B<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"30%"</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"50%"</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span>
<span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>C<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="o">=</span><span class="s">"row"</span> <span class="na">fxLayoutGap</span><span class="o">=</span><span class="s">"10px"</span> <span class="na">fxFlex</span><span class="o">=</span><span class="s">"33%"</span> <span class="na">class</span><span class="o">=</span><span class="s">"block-3"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"30%"</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"50%"</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span>
<span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>D<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">gt-sm</span><span class="o">=</span><span class="s">"70%"</span> <span class="na">fxFlex</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"50%"</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span>
<span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>E<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">container</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">height</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">vh</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">.</span><span class="nc">grid-item</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="kc">lightsalmon</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">border-radius</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">height</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<h3 style="border-bottom: 0.18rem solid black;">Layout 5 - Holy Grail Layout</h3>
<p>This is a responsive holy grail layout.</p>
<p><img alt="Layout 5 - Holy Grail Layout" src="/images/flexlayout-layout5.png"></p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="o">=</span><span class="s">"column"</span> <span class="na">fxLayoutGap</span><span class="o">=</span><span class="s">"10px"</span> <span class="na">class</span><span class="o">=</span><span class="s">"container"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="o">=</span><span class="s">"25%"</span> <span class="na">class</span><span class="o">=</span><span class="s">"block-1"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span> <span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>A<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxLayout</span><span class="o">=</span><span class="s">"row"</span> <span class="na">fxLayout</span><span class="err">.</span><span class="na">lt-md</span><span class="o">=</span><span class="s">"column"</span> <span class="na">fxLayoutGap</span><span class="o">=</span><span class="s">"10px"</span> <span class="na">fxFlex</span><span class="o">=</span><span class="s">"50%"</span> <span class="na">class</span><span class="o">=</span><span class="s">"block-2"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span> <span class="na">fxHide</span><span class="err">.</span><span class="na">lt-md</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span>
<span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>B<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="o">=</span><span class="s">"75%"</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span>
<span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>C<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span>
<span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>D<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span><span class="o">=</span><span class="s">"25%"</span> <span class="na">class</span><span class="o">=</span><span class="s">"block-3"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">fxFlex</span> <span class="na">fxLayoutAlign</span><span class="o">=</span><span class="s">"center center"</span> <span class="na">fxFlexFill</span> <span class="na">class</span><span class="o">=</span><span class="s">"grid-item"</span><span class="p">></span>E<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">container</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">height</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">vh</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">.</span><span class="nc">grid-item</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="kc">lightsalmon</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">border-radius</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">height</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Hope this helps!</p>Customizing AWS Amplify color scheme for Angular2021-05-13T16:00:00+05:302021-05-13T16:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-05-13:/blog/customizing-aws-amplify-color-scheme-for-angular.html<p>In this article, I'll show you how to customize <a href="https://aws.amazon.com/amplify">AWS Amplify</a> color scheme for <a href="https://angular.io">Angular</a> apps.</p>
<p>Following are the list of variables with the default values exposed by AWS Amplify for customization.</p>
<div class="highlight"><pre><span></span><code><span class="p">:</span><span class="nd">root</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nv">--amplify-primary-color</span><span class="p">:</span><span class="w"> </span><span class="mh">#ff9900</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-primary-contrast</span><span class="o">:</span><span class="w"> </span><span class="nf">var</span><span class="p">(</span><span class="err">–</span><span class="n">amplify</span><span class="nv">-white</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-primary-tint</span><span class="o">:</span><span class="w"> </span><span class="mh">#ffac31</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-primary-shade</span><span class="o">:</span><span class="w"> </span><span class="mh">#e88b01</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-secondary-color</span><span class="o">:</span><span class="w"> </span><span class="mh">#152939</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-secondary-contrast</span><span class="o">:</span><span class="w"> </span><span class="nf">var</span><span class="p">(</span><span class="err">–</span><span class="n">amplify</span><span class="nv">-white</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-secondary-tint …</span></code></pre></div><p>In this article, I'll show you how to customize <a href="https://aws.amazon.com/amplify">AWS Amplify</a> color scheme for <a href="https://angular.io">Angular</a> apps.</p>
<p>Following are the list of variables with the default values exposed by AWS Amplify for customization.</p>
<div class="highlight"><pre><span></span><code><span class="p">:</span><span class="nd">root</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nv">--amplify-primary-color</span><span class="p">:</span><span class="w"> </span><span class="mh">#ff9900</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-primary-contrast</span><span class="o">:</span><span class="w"> </span><span class="nf">var</span><span class="p">(</span><span class="err">–</span><span class="n">amplify</span><span class="nv">-white</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-primary-tint</span><span class="o">:</span><span class="w"> </span><span class="mh">#ffac31</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-primary-shade</span><span class="o">:</span><span class="w"> </span><span class="mh">#e88b01</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-secondary-color</span><span class="o">:</span><span class="w"> </span><span class="mh">#152939</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-secondary-contrast</span><span class="o">:</span><span class="w"> </span><span class="nf">var</span><span class="p">(</span><span class="err">–</span><span class="n">amplify</span><span class="nv">-white</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-secondary-tint</span><span class="o">:</span><span class="w"> </span><span class="mh">#31465f</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-secondary-shade</span><span class="o">:</span><span class="w"> </span><span class="mh">#1F2A37</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-tertiary-color</span><span class="o">:</span><span class="w"> </span><span class="mh">#5d8aff</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-tertiary-contrast</span><span class="o">:</span><span class="w"> </span><span class="nf">var</span><span class="p">(</span><span class="err">–</span><span class="n">amplify</span><span class="nv">-white</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-tertiary-tint</span><span class="o">:</span><span class="w"> </span><span class="mh">#7da1ff</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-tertiary-shade</span><span class="o">:</span><span class="w"> </span><span class="mh">#537BE5</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-grey</span><span class="o">:</span><span class="w"> </span><span class="mh">#828282</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-light-grey</span><span class="o">:</span><span class="w"> </span><span class="mh">#c4c4c4</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-white</span><span class="o">:</span><span class="w"> </span><span class="mh">#ffffff</span><span class="w"></span>
<span class="w"> </span><span class="o">--</span><span class="n">amplify-red</span><span class="o">:</span><span class="w"> </span><span class="mh">#dd3f5b</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>You can overwrite these variables with your custom colors by defining <code>:root</code> class in <code>styles.scss</code> file.</p>
<p>Hope this helps!</p>Dynamically Change Page Title for each Route in Angular2021-04-29T14:00:00+05:302021-04-29T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-04-29:/blog/dynamically-change-page-title-route-in-angular.html<p>In this article, I'll show you how to dynamically change page title of an <a href="https://angular.io">Angular</a> app based on Route configuration.</p>
<p>Let say we have three components <code>HomeComponent</code>, <code>MainComponent</code> and <code>AboutComponent</code>. The corresponding route definition as in <code>app-routing.module.ts</code> file </p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">NgModule</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@angular/core"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">RouterModule</span><span class="p">,</span><span class="w"> </span><span class="nx">Routes</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@angular …</span></code></pre></div><p>In this article, I'll show you how to dynamically change page title of an <a href="https://angular.io">Angular</a> app based on Route configuration.</p>
<p>Let say we have three components <code>HomeComponent</code>, <code>MainComponent</code> and <code>AboutComponent</code>. The corresponding route definition as in <code>app-routing.module.ts</code> file </p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">NgModule</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@angular/core"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">RouterModule</span><span class="p">,</span><span class="w"> </span><span class="nx">Routes</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@angular/router"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">AboutComponent</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"./about/about.component"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">HomeComponent</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"./home/home.component"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">MainComponent</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"./main/main.component"</span><span class="p">;</span><span class="w"></span>
<span class="kd">const</span><span class="w"> </span><span class="nx">routes</span><span class="o">:</span><span class="w"> </span><span class="kt">Routes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">path</span><span class="o">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w"> </span><span class="nx">redirectTo</span><span class="o">:</span><span class="w"> </span><span class="s2">"home"</span><span class="p">,</span><span class="w"> </span><span class="nx">pathMatch</span><span class="o">:</span><span class="w"> </span><span class="s2">"full"</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">path</span><span class="o">:</span><span class="w"> </span><span class="s2">"home"</span><span class="p">,</span><span class="w"> </span><span class="nx">component</span><span class="o">:</span><span class="w"> </span><span class="kt">HomeComponent</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">title</span><span class="o">:</span><span class="w"> </span><span class="s2">"Home Page"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">path</span><span class="o">:</span><span class="w"> </span><span class="s2">"main"</span><span class="p">,</span><span class="w"> </span><span class="nx">component</span><span class="o">:</span><span class="w"> </span><span class="kt">MainComponent</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">title</span><span class="o">:</span><span class="w"> </span><span class="s2">"Main Page"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">path</span><span class="o">:</span><span class="w"> </span><span class="s2">"about"</span><span class="p">,</span><span class="w"> </span><span class="nx">component</span><span class="o">:</span><span class="w"> </span><span class="kt">AboutComponent</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">];</span><span class="w"></span>
<span class="kd">@NgModule</span><span class="p">({</span><span class="w"></span>
<span class="w"> </span><span class="nx">imports</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="nx">RouterModule</span><span class="p">.</span><span class="nx">forRoot</span><span class="p">(</span><span class="nx">routes</span><span class="p">)],</span><span class="w"></span>
<span class="w"> </span><span class="nx">exports</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="nx">RouterModule</span><span class="p">]</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
<span class="k">export</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nx">AppRoutingModule</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
</code></pre></div>
<p>Here we have defined custom title for <code>HomeComponent</code> and <code>MainComponent</code> in the data section.</p>
<p>In <code>app.component.ts</code> file we need to subscribe for router events and populate the page title</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">Component</span><span class="p">,</span><span class="w"> </span><span class="nx">OnInit</span><span class="p">,</span><span class="w"> </span><span class="nx">VERSION</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@angular/core"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">Title</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@angular/platform-browser"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">ActivatedRoute</span><span class="p">,</span><span class="w"> </span><span class="nx">NavigationEnd</span><span class="p">,</span><span class="w"> </span><span class="nx">Router</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@angular/router"</span><span class="p">;</span><span class="w"></span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">filter</span><span class="p">,</span><span class="w"> </span><span class="nx">map</span><span class="p">,</span><span class="w"> </span><span class="nx">mergeMap</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"rxjs/operators"</span><span class="p">;</span><span class="w"></span>
<span class="kd">@Component</span><span class="p">({</span><span class="w"></span>
<span class="w"> </span><span class="nx">selector</span><span class="o">:</span><span class="w"> </span><span class="s2">"my-app"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">templateUrl</span><span class="o">:</span><span class="w"> </span><span class="s2">"./app.component.html"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">styleUrls</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"./app.component.css"</span><span class="p">]</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
<span class="k">export</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nx">AppComponent</span><span class="w"> </span><span class="k">implements</span><span class="w"> </span><span class="nx">OnInit</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"Angular "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">VERSION</span><span class="p">.</span><span class="nx">major</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kr">constructor</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">private</span><span class="w"> </span><span class="nx">router</span><span class="o">:</span><span class="w"> </span><span class="kt">Router</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">private</span><span class="w"> </span><span class="nx">activatedRoute</span><span class="o">:</span><span class="w"> </span><span class="kt">ActivatedRoute</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">private</span><span class="w"> </span><span class="nx">titleService</span><span class="o">:</span><span class="w"> </span><span class="kt">Title</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
<span class="w"> </span><span class="nx">ngOnInit</span><span class="p">()</span><span class="o">:</span><span class="w"> </span><span class="ow">void</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">router</span><span class="p">.</span><span class="nx">events</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">pipe</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">filter</span><span class="p">(</span><span class="nx">event</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">event</span><span class="w"> </span><span class="ow">instanceof</span><span class="w"> </span><span class="nx">NavigationEnd</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nx">map</span><span class="p">(()</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">route</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">activatedRoute</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="nx">route</span><span class="p">.</span><span class="nx">firstChild</span><span class="p">)</span><span class="w"> </span><span class="nx">route</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">route</span><span class="p">.</span><span class="nx">firstChild</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">route</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}),</span><span class="w"></span>
<span class="w"> </span><span class="nx">filter</span><span class="p">((</span><span class="nx">route</span><span class="o">:</span><span class="w"> </span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">route</span><span class="p">.</span><span class="nx">outlet</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"primary"</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nx">mergeMap</span><span class="p">((</span><span class="nx">route</span><span class="o">:</span><span class="w"> </span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">route</span><span class="p">.</span><span class="nx">data</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nx">map</span><span class="p">((</span><span class="nx">data</span><span class="o">:</span><span class="w"> </span><span class="kt">any</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">title</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">data</span><span class="p">.</span><span class="nx">title</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="s2">"Default App Title"</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">subscribe</span><span class="p">(</span><span class="nx">pathString</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">titleService</span><span class="p">.</span><span class="nx">setTitle</span><span class="p">(</span><span class="nx">pathString</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Define navigations in <code>app.component.html</code> file as</p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">a</span> <span class="na">routerLink</span><span class="o">=</span><span class="s">'home'</span><span class="p">></span>Home<span class="p"></</span><span class="nt">a</span><span class="p">></span><span class="ni">&nbsp;</span>
<span class="p"><</span><span class="nt">a</span> <span class="na">routerLink</span><span class="o">=</span><span class="s">'main'</span><span class="p">></span>Main<span class="p"></</span><span class="nt">a</span><span class="p">></span><span class="ni">&nbsp;</span>
<span class="p"><</span><span class="nt">a</span> <span class="na">routerLink</span><span class="o">=</span><span class="s">'about'</span><span class="p">></span>About<span class="p"></</span><span class="nt">a</span><span class="p">></span>
<span class="p"><</span><span class="nt">router-outlet</span><span class="p">></</span><span class="nt">router-outlet</span><span class="p">></span>
</code></pre></div>
<p>As you navigate the pages, you can see the page title as set in <code>app-routing.module.ts</code> file.</p>
<p>Hope this helps!</p>Multiple head revisions present ERROR in Flask Migrate2021-04-27T18:00:00+05:302021-04-27T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-04-27:/blog/multiple-head-revisions-present-error-flask-migrate.html<p>In this article, I'll show you the cause of the following error and how to fix it.</p>
<div class="highlight"><pre><span></span><code>ERROR [root] Error: Multiple head revisions are present for given argument 'head';
please specify a specific target revision, '<branchname>@head' to narrow
to a specific head, or 'heads' for all heads
</code></pre></div>
<p>This error …</p><p>In this article, I'll show you the cause of the following error and how to fix it.</p>
<div class="highlight"><pre><span></span><code>ERROR [root] Error: Multiple head revisions are present for given argument 'head';
please specify a specific target revision, '<branchname>@head' to narrow
to a specific head, or 'heads' for all heads
</code></pre></div>
<p>This error will showup whenever you run Flask Migrate upgrade migration command using</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>venv<span class="o">)</span> $ flask db upgrade
</code></pre></div>
<p><a href="https://flask-migrate.readthedocs.io/en/latest">Flask Migrate</a> is an extension that handles <a href="https://www.sqlalchemy.org">SQLAlchemy</a> database migrations for <a href="https://flask.palletsprojects.com">Flask</a> applications using <a href="https://alembic.sqlalchemy.org/en/latest/index.html">Alembic</a></p>
<p>We'll demystify when this error will occur with an example scenario. Let the initial migraion state be</p>
<div class="highlight"><pre><span></span><code>* C (head)
|
* B
|
* A
</code></pre></div>
<p>Say, two developers create a migration branches from <code>C</code></p>
<div class="highlight"><pre><span></span><code>* F
|
| * E
| |
| * D
|/
* C (head)
|
* B
|
* A
</code></pre></div>
<p>One developer pushes <code>F</code> to become new <code>head</code></p>
<div class="highlight"><pre><span></span><code>* F (head)
|
| * E (head)
| |
| * D
|/
* C
|
* B
|
* A
</code></pre></div>
<p>Now if another developer merges the changes and run <code>flask db upgrade</code> will end up having the above ERROR.</p>
<p>This inconsistency state can be identified using the command</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>venv<span class="o">)</span> $ flask db <span class="nb">history</span>
</code></pre></div>
<p>To see the following output, where we can see it clearly there are two <code>head</code> revisions present.</p>
<div class="highlight"><pre><span></span><code>D -> E (head), empty message
C -> D, empty message
C -> F (head), empty message
B -> C (branchpoint), empty message
A -> B, empty message
<base> -> A, empty message
</code></pre></div>
<p>Now, you can manually resolve this conflict by modifying the <code>down_revision</code> or using the following command</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>venv<span class="o">)</span> $ flask db merge heads -m <span class="s2">"merging two heads"</span>
</code></pre></div>
<p>You can even explicitly mention which two heads you wanted to merge like</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>venv<span class="o">)</span> $ flask db merge D F -m <span class="s2">"merging D and F"</span>
</code></pre></div>
<p>Hope this helps!</p>Python Pipenv quick start guide for managing packages and virtualenv2021-04-23T18:00:00+05:302021-04-23T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-04-23:/blog/python-pipenv-quick-start-packages-virtualenv.html<p>In this video I'm going to show you how to start using pipenv for your python projects.</p>
<p><a href="https://pipenv.pypa.io">Pipenv</a> is a packaging tool which automatically creates and manages virtualenv and packages.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/H_9LMf_6-vE" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<div class="highlight"><pre><span></span><code><span class="c1"># To create a new virtual environment</span>
$ mkdir pipenv-demo <span class="o">&&</span> <span class="nb">cd</span> pipenv-demo
$ pipenv shell
<span class="o">(</span>venv<span class="o">)</span> $ cat Pipfile
<span class="c1"># To install a package and …</span></code></pre></div><p>In this video I'm going to show you how to start using pipenv for your python projects.</p>
<p><a href="https://pipenv.pypa.io">Pipenv</a> is a packaging tool which automatically creates and manages virtualenv and packages.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/H_9LMf_6-vE" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<div class="highlight"><pre><span></span><code><span class="c1"># To create a new virtual environment</span>
$ mkdir pipenv-demo <span class="o">&&</span> <span class="nb">cd</span> pipenv-demo
$ pipenv shell
<span class="o">(</span>venv<span class="o">)</span> $ cat Pipfile
<span class="c1"># To install a package and add to Pipfile</span>
<span class="o">(</span>venv<span class="o">)</span> $ pipenv install flask
<span class="c1"># To install a dev package</span>
<span class="o">(</span>venv<span class="o">)</span> $ pipenv install pytest --dev
<span class="c1"># To install all packages from Pipfile</span>
<span class="o">(</span>venv<span class="o">)</span> $ pipenv install
<span class="c1"># To install all packages including dev-packages</span>
<span class="o">(</span>venv<span class="o">)</span> $ pipenv install --dev
<span class="c1"># To show the dependency graph</span>
<span class="o">(</span>venv<span class="o">)</span> $ pipenv graph
<span class="c1"># To uninstall a package and remove from Pipfile</span>
<span class="o">(</span>venv<span class="o">)</span> $ pipenv uninstall pytest
</code></pre></div>
<p>Hope this helps!</p>Using PyCharm Debugger with Chalice for Local Debugging in Windows2021-04-10T14:00:00+05:302021-04-10T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-04-10:/blog/pycharm-for-debugging-chalice-windows.html<p>In this video, I'll show how to use <a href="https://www.jetbrains.com/pycharm">PyCharm</a> Debugger to debug Chalice application in Windows.</p>
<p><a href="https://aws.github.io/chalice">Chalice</a> is a framework develop Serverless application in Python, so that you can quickly create and deploy applications that use various <a href="https://aws.amazon.com">AWS</a> services like: Lambda, API Gateway etc.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/5yz5VZXDGDY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<div class="highlight"><pre><span></span><code>$ mkdir chalice-debug <span class="o">&&</span> <span class="nb">cd</span> chalice-debug
$ python -m …</code></pre></div><p>In this video, I'll show how to use <a href="https://www.jetbrains.com/pycharm">PyCharm</a> Debugger to debug Chalice application in Windows.</p>
<p><a href="https://aws.github.io/chalice">Chalice</a> is a framework develop Serverless application in Python, so that you can quickly create and deploy applications that use various <a href="https://aws.amazon.com">AWS</a> services like: Lambda, API Gateway etc.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/5yz5VZXDGDY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<div class="highlight"><pre><span></span><code>$ mkdir chalice-debug <span class="o">&&</span> <span class="nb">cd</span> chalice-debug
$ python -m venv venv
$ venv<span class="se">\S</span>cripts<span class="se">\a</span>ctivate.bat
<span class="o">(</span>venv<span class="o">)</span> $ pip install chalice
<span class="o">(</span>venv<span class="o">)</span> $ chalice new-project chalice-debug
<span class="o">(</span>venv<span class="o">)</span> $ <span class="nb">cd</span> chalice-debug
<span class="o">(</span>venv<span class="o">)</span> $ chalice <span class="nb">local</span> --no-autoreload
</code></pre></div>
<p>In Pycharm IDE, <code>Run</code> -> <code>Attach To Process</code> and select the chalice process to attach and start debugging</p>
<p>Hope this helps!</p>Using SQLAlchemy Expression for Partial JSON Update2021-01-21T14:00:00+05:302021-01-21T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2021-01-21:/blog/sqlalchemy-expression-partial-json-update.html<p>In this article, I'll show how to partially update JSON column using SQLAlchemy expressions in MySQL database.</p>
<p>Say, we have a table named <code>TableName</code> with a JSON column named <code>JsonColumn</code> and the values in this JSON column is in the following format;</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"JsonAttribute"</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"AnotherJsonAttribute"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Hello"</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>SQL statement to update …</p><p>In this article, I'll show how to partially update JSON column using SQLAlchemy expressions in MySQL database.</p>
<p>Say, we have a table named <code>TableName</code> with a JSON column named <code>JsonColumn</code> and the values in this JSON column is in the following format;</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"JsonAttribute"</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"AnotherJsonAttribute"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Hello"</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>SQL statement to update a single JSON attribute is as below;</p>
<div class="highlight"><pre><span></span><code><span class="k">UPDATE</span><span class="w"> </span><span class="n">TableName</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">JsonColumn</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">JSON_SET</span><span class="p">(</span><span class="n">JsonColumn</span><span class="p">,</span><span class="w"> </span><span class="s1">'$.JsonAttribute'</span><span class="p">,</span><span class="w"> </span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">Id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Converting the above SQL statement to the corresponding SQLAlchemy expression is as below;</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">update</span><span class="p">,</span> <span class="n">func</span>
<span class="n">table_name_id</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">json_attribute_value</span> <span class="o">=</span> <span class="mi">10</span>
<span class="n">update_sql</span> <span class="o">=</span> <span class="n">update</span><span class="p">(</span><span class="n">TableName</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="p">(</span>
<span class="n">JsonColumn</span><span class="o">=</span><span class="n">func</span><span class="o">.</span><span class="n">json_set</span><span class="p">(</span><span class="n">TableName</span><span class="o">.</span><span class="n">JsonColumn</span><span class="p">,</span> <span class="s1">'$.JsonAttribute'</span><span class="p">,</span> <span class="n">json_attribute_value</span><span class="p">))</span><span class="o">.</span>\
<span class="n">where</span><span class="p">(</span><span class="n">TableName</span><span class="o">.</span><span class="n">Id</span> <span class="o">==</span> <span class="n">table_name_id</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">update_sql</span><span class="p">)</span>
</code></pre></div>
<p>For running a bulk update on the array of dicts using <code>WHERE</code> condition.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">update</span><span class="p">,</span> <span class="n">func</span><span class="p">,</span> <span class="n">bindparam</span>
<span class="n">update_data</span> <span class="o">=</span> <span class="p">[{</span>
<span class="s1">'_Id'</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span>
<span class="s1">'Value'</span><span class="p">:</span> <span class="mi">10</span>
<span class="p">},</span> <span class="p">{</span>
<span class="s1">'_Id'</span><span class="p">:</span> <span class="mi">101</span><span class="p">,</span>
<span class="s1">'Value'</span><span class="p">:</span> <span class="mi">11</span>
<span class="p">},</span> <span class="p">{</span>
<span class="s1">'_Id'</span><span class="p">:</span> <span class="mi">102</span><span class="p">,</span>
<span class="s1">'Value'</span><span class="p">:</span> <span class="mi">12</span>
<span class="p">}]</span>
<span class="n">update_sql</span> <span class="o">=</span> <span class="n">update</span><span class="p">(</span><span class="n">TableName</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="p">(</span>
<span class="n">JsonColumn</span><span class="o">=</span><span class="n">func</span><span class="o">.</span><span class="n">json_set</span><span class="p">(</span><span class="n">TableName</span><span class="o">.</span><span class="n">JsonColumn</span><span class="p">,</span> <span class="s1">'$.JsonAttribute'</span><span class="p">,</span> <span class="n">bindparam</span><span class="p">(</span><span class="s1">'Value'</span><span class="p">)))</span><span class="o">.</span>\
<span class="n">where</span><span class="p">(</span><span class="n">TableName</span><span class="o">.</span><span class="n">Id</span> <span class="o">==</span> <span class="n">bindparam</span><span class="p">(</span><span class="s1">'_Id'</span><span class="p">))</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">update_sql</span><span class="p">,</span> <span class="n">update_data</span><span class="p">)</span>
</code></pre></div>
<p>Hope this helps!</p>Definitive guide to solve CORS error, Access-Control-Allow-Origin in Python Flask APIs2020-10-02T18:00:00+05:302020-10-02T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2020-10-02:/blog/definitive-guide-to-solve-cors-access-control-allow-origin-python-flask.html<p>In this video, I'll show how to enable <a href="https://flask-cors.readthedocs.io">Flask-CORS</a> for <a href="https://flask.palletsprojects.com">Flask</a> based API projects. Explains and code three different options available to configure CORS for your projects.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/-u5BdG_6Lzs" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Starting with the basic Flask project.</p>
<ul>
<li>Update <code>requirements.txt</code> file to include <code>Flask-CORS</code> and install the requirements</li>
</ul>
<div class="highlight"><pre><span></span><code>Flask
Flask-CORS
</code></pre></div>
<h3>Option 1: Configuring globally …</h3><p>In this video, I'll show how to enable <a href="https://flask-cors.readthedocs.io">Flask-CORS</a> for <a href="https://flask.palletsprojects.com">Flask</a> based API projects. Explains and code three different options available to configure CORS for your projects.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/-u5BdG_6Lzs" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Starting with the basic Flask project.</p>
<ul>
<li>Update <code>requirements.txt</code> file to include <code>Flask-CORS</code> and install the requirements</li>
</ul>
<div class="highlight"><pre><span></span><code>Flask
Flask-CORS
</code></pre></div>
<h3>Option 1: Configuring globally</h3>
<ul>
<li>Enable CORS globally, with permissive configurations</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">CORS</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
</code></pre></div>
<p><code>curl</code> command to test the configuration</p>
<div class="highlight"><pre><span></span><code>$ curl -v -X OPTIONS
http://localhost:5000/hello
</code></pre></div>
<ul>
<li>Configure <code>Access-Control-Allow-Origin</code> option</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">api_v1_cors_config</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"origins"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"http://localhost:5000"</span><span class="p">]</span>
<span class="p">}</span>
<span class="n">CORS</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">resources</span><span class="o">=</span><span class="p">{</span><span class="s2">"/api/v1/*"</span><span class="p">:</span> <span class="n">api_v1_cors_config</span><span class="p">})</span>
</code></pre></div>
<p><code>curl</code> command to test the configuration</p>
<div class="highlight"><pre><span></span><code>$ curl -v -X OPTIONS
-H <span class="s2">"Origin: http://localhost:5000"</span>
http://localhost:5000/api/v1/hello
</code></pre></div>
<ul>
<li>Configure <code>Access-Control-Allow-Methods</code> option</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">api_v1_cors_config</span> <span class="o">=</span> <span class="p">{</span>
<span class="o">...</span>
<span class="s2">"methods"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"OPTIONS"</span><span class="p">,</span> <span class="s2">"GET"</span><span class="p">,</span> <span class="s2">"POST"</span><span class="p">],</span>
<span class="p">}</span>
<span class="n">CORS</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">resources</span><span class="o">=</span><span class="p">{</span><span class="s2">"/api/v1/*"</span><span class="p">:</span> <span class="n">api_v1_cors_config</span><span class="p">})</span>
</code></pre></div>
<p><code>curl</code> command to test the configuration</p>
<div class="highlight"><pre><span></span><code>$ curl -v -X OPTIONS
-H <span class="s2">"Origin: http://localhost:5000"</span>
-H <span class="s2">"Access-Control-Request-Method: GET"</span>
http://localhost:5000/api/v1/hello
</code></pre></div>
<ul>
<li>Configure <code>Access-Control-Allow-Headers</code> option</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">api_v1_cors_config</span> <span class="o">=</span> <span class="p">{</span>
<span class="o">...</span>
<span class="s2">"allow_headers"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"Authorization"</span><span class="p">]</span>
<span class="p">}</span>
<span class="n">CORS</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">resources</span><span class="o">=</span><span class="p">{</span><span class="s2">"/api/v1/*"</span><span class="p">:</span> <span class="n">api_v1_cors_config</span><span class="p">})</span>
</code></pre></div>
<p><code>curl</code> command to test the configuration</p>
<div class="highlight"><pre><span></span><code>$ curl -v -X OPTIONS
-H <span class="s2">"Origin: http://localhost:5000"</span>
-H <span class="s2">"Access-Control-Request-Method: GET"</span>
-H <span class="s2">"Access-Control-Request-Headers: Authorization"</span>
http://localhost:5000/api/v1/hello
</code></pre></div>
<h3>Option 2: Configuring using Decorator</h3>
<div class="highlight"><pre><span></span><code><span class="n">api_v2_cors_config</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"origins"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"http://localhost:5000"</span><span class="p">],</span>
<span class="s2">"methods"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"OPTIONS"</span><span class="p">,</span> <span class="s2">"GET"</span><span class="p">,</span> <span class="s2">"POST"</span><span class="p">],</span>
<span class="s2">"allow_headers"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"Authorization"</span><span class="p">,</span> <span class="s2">"Content-Type"</span><span class="p">]</span>
<span class="p">}</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s1">'/api/v2/hello'</span><span class="p">)</span>
<span class="nd">@cross_origin</span><span class="p">(</span><span class="o">**</span><span class="n">api_v2_cors_config</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">v2_hello</span><span class="p">():</span>
<span class="k">return</span> <span class="s1">'Hello World v2'</span>
</code></pre></div>
<h3>Option 3: Configuring globally for multiple resources</h3>
<div class="highlight"><pre><span></span><code><span class="n">CORS</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">resources</span><span class="o">=</span><span class="p">{</span>
<span class="sa">r</span><span class="s2">"/api/v1/*"</span><span class="p">:</span> <span class="n">api_v1_cors_config</span><span class="p">,</span>
<span class="sa">r</span><span class="s2">"/api/v2/*"</span><span class="p">:</span> <span class="n">api_v2_cors_config</span>
<span class="p">})</span>
</code></pre></div>
<p>Hope this helps!</p>How to create and run Angular app using Docker without installing Node in the Host machine2020-09-19T18:00:00+05:302020-09-19T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2020-09-19:/blog/how-to-run-angular-using-docker-without-installing-node-host-machine.html<p>In this video, I'll show how to create and run <a href="https://angular.io">Angular</a> app using <a href="https://www.docker.com">Docker</a>, without installing <a href="https://nodejs.org">Node</a> in the Host machine.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/e4n8Dja5PKo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<ul>
<li>Create folder</li>
</ul>
<div class="highlight"><pre><span></span><code>$ mkdir ng-docker <span class="o">&&</span> <span class="nb">cd</span> ng-docker
</code></pre></div>
<ul>
<li>Create <code>Dockerfile</code> file</li>
</ul>
<div class="highlight"><pre><span></span><code>FROM node:12-alpine
LABEL <span class="nv">AUTHOR</span><span class="o">=</span><span class="s2">"Arunkumar Dharuman"</span>
WORKDIR /code
RUN npm install -g @angular/cli
</code></pre></div>
<ul>
<li>Create <code>docker-compose.yaml</code> file</li>
</ul>
<div class="highlight"><pre><span></span><code>version …</code></pre></div><p>In this video, I'll show how to create and run <a href="https://angular.io">Angular</a> app using <a href="https://www.docker.com">Docker</a>, without installing <a href="https://nodejs.org">Node</a> in the Host machine.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/e4n8Dja5PKo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<ul>
<li>Create folder</li>
</ul>
<div class="highlight"><pre><span></span><code>$ mkdir ng-docker <span class="o">&&</span> <span class="nb">cd</span> ng-docker
</code></pre></div>
<ul>
<li>Create <code>Dockerfile</code> file</li>
</ul>
<div class="highlight"><pre><span></span><code>FROM node:12-alpine
LABEL <span class="nv">AUTHOR</span><span class="o">=</span><span class="s2">"Arunkumar Dharuman"</span>
WORKDIR /code
RUN npm install -g @angular/cli
</code></pre></div>
<ul>
<li>Create <code>docker-compose.yaml</code> file</li>
</ul>
<div class="highlight"><pre><span></span><code>version: <span class="s2">"3.8"</span>
services:
app:
build: .
restart: always
volumes:
- <span class="s2">"/code/node_modules"</span>
- ./:/code
ports:
- <span class="m">4200</span>:4200
commands:
ng serve --host <span class="m">0</span>.0.0.0
</code></pre></div>
<ul>
<li>Create Angular project without installing packages</li>
</ul>
<div class="highlight"><pre><span></span><code>$ docker-compose run app ng new ng-docker --directory . --skipInstall
</code></pre></div>
<ul>
<li>Update <code>Dockerfile</code> file and append following lines</li>
</ul>
<div class="highlight"><pre><span></span><code>RUN npm install -g @angular/cli
COPY package*.json ./
RUN npm install
</code></pre></div>
<ul>
<li>Run <code>docker-compose</code></li>
</ul>
<div class="highlight"><pre><span></span><code>$ docker-compose up --build
</code></pre></div>
<ul>
<li>Further, from next run</li>
</ul>
<div class="highlight"><pre><span></span><code>$ docker-compose up
</code></pre></div>
<p>Hope this helps!</p>How to use Angular CLI and self-signed SSL Certificate to serve over HTTPS in localhost2020-09-10T18:00:00+05:302020-09-10T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2020-09-10:/blog/angular-cli-ssl-https-localhost.html<p>In this video I'll show how create a self-signed SSL certificate and configure it on Windows 10 machine. Using Angular CLI and self-signed SSL Certificate to serve application over HTTPS localhost.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/OWjVXb7-mSU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Hope this helps!</p>News article summarization with Python NLTK2020-02-06T18:00:00+05:302020-02-06T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2020-02-06:/blog/news-article-summarization-with-NLTK.html<p>In this post I will share the code to summarize a news article using Python's <a href="https://www.nltk.org/">Natural Language Toolkit</a> (NLTK)</p>
<p>For this example I'll be extracting an article from <a href="https://www.thehindu.com/opinion/editorial/purifying-water-the-hindu-editorial-on-draft-notification-on-ro-systems/article30745293.ece">The Hindu</a> using <a href="https://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a> and summarize the article using word frequency distribution. </p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="kn">from</span> <span class="nn">nltk.tokenize</span> <span class="kn">import …</span></code></pre></div><p>In this post I will share the code to summarize a news article using Python's <a href="https://www.nltk.org/">Natural Language Toolkit</a> (NLTK)</p>
<p>For this example I'll be extracting an article from <a href="https://www.thehindu.com/opinion/editorial/purifying-water-the-hindu-editorial-on-draft-notification-on-ro-systems/article30745293.ece">The Hindu</a> using <a href="https://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a> and summarize the article using word frequency distribution. </p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="kn">from</span> <span class="nn">nltk.tokenize</span> <span class="kn">import</span> <span class="n">sent_tokenize</span><span class="p">,</span> <span class="n">word_tokenize</span>
<span class="kn">from</span> <span class="nn">nltk.corpus</span> <span class="kn">import</span> <span class="n">stopwords</span>
<span class="kn">from</span> <span class="nn">string</span> <span class="kn">import</span> <span class="n">punctuation</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="n">article_url</span> <span class="o">=</span> <span class="s1">'https://www.thehindu.com/opinion/editorial/purifying-water-the-hindu-editorial-on-draft-notification-on-ro-systems/article30745293.ece'</span>
<span class="c1"># Extracting the content from html page</span>
<span class="k">def</span> <span class="nf">get_article_text</span><span class="p">(</span><span class="n">url</span><span class="p">):</span>
<span class="n">page</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">article_url</span><span class="p">)</span>
<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">page</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="n">features</span><span class="o">=</span><span class="s1">'lxml'</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">soup</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'div'</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s1">'content-body'</span><span class="p">))</span>
<span class="k">return</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">p</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">content</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="s1">'p'</span><span class="p">)])</span>
<span class="n">content_text</span> <span class="o">=</span> <span class="n">get_article_text</span><span class="p">(</span><span class="n">article_url</span><span class="p">)</span>
<span class="c1"># summarize the article with 'n' number of sentences</span>
<span class="k">def</span> <span class="nf">summarize</span><span class="p">(</span><span class="n">content_text</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span>
<span class="n">sent_tokens</span> <span class="o">=</span> <span class="n">sent_tokenize</span><span class="p">(</span><span class="n">content_text</span><span class="p">)</span>
<span class="n">word_tokens</span> <span class="o">=</span> <span class="n">word_tokenize</span><span class="p">(</span><span class="n">content_text</span><span class="p">)</span>
<span class="n">_stop_words</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">stopwords</span><span class="o">.</span><span class="n">words</span><span class="p">(</span><span class="s1">'english'</span><span class="p">)</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">punctuation</span><span class="p">))</span>
<span class="n">word_tokens_wo_stopwords</span> <span class="o">=</span> <span class="p">[</span><span class="n">word</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_tokens</span> <span class="k">if</span> <span class="n">word</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">_stop_words</span><span class="p">]</span>
<span class="c1"># Frequency distribution</span>
<span class="kn">from</span> <span class="nn">nltk.probability</span> <span class="kn">import</span> <span class="n">FreqDist</span>
<span class="kn">from</span> <span class="nn">heapq</span> <span class="kn">import</span> <span class="n">nlargest</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="n">freq</span> <span class="o">=</span> <span class="n">FreqDist</span><span class="p">(</span><span class="n">word_tokens_wo_stopwords</span><span class="p">)</span>
<span class="n">nlargest</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">freq</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">freq</span><span class="o">.</span><span class="n">get</span><span class="p">)</span>
<span class="n">ranking</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">sent</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">sent_tokens</span><span class="p">):</span>
<span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">word_tokenize</span><span class="p">(</span><span class="n">sent</span><span class="o">.</span><span class="n">lower</span><span class="p">()):</span>
<span class="k">if</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">freq</span><span class="p">:</span>
<span class="n">ranking</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">+=</span> <span class="n">freq</span><span class="p">[</span><span class="n">w</span><span class="p">]</span>
<span class="n">sent_idx</span> <span class="o">=</span> <span class="n">nlargest</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">ranking</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">ranking</span><span class="o">.</span><span class="n">get</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">sent_tokens</span><span class="p">[</span><span class="n">j</span><span class="p">]</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">sent_idx</span><span class="p">)]</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">summarize</span><span class="p">(</span><span class="n">content_text</span><span class="p">,</span> <span class="mi">3</span><span class="p">)))</span>
</code></pre></div>
<p>Final output looks like this;
<img alt="Output" src="/images/news_summarization.png"></p>
<p>Hope this helps!</p>Progressively adding table rows with Angular2020-01-31T14:00:00+05:302020-01-31T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2020-01-31:/blog/progressively-adding-table-rows-with-angular.html<p>I will show how to progressively add new rows to the table as the last row is being edited in <a href="https://angular.io">Angular</a></p>
<p>In, <code>app.component.html</code></p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">table</span> <span class="na">class</span><span class="o">=</span><span class="s">"progressive-table"</span><span class="p">></span>
<span class="p"><</span><span class="nt">thead</span><span class="p">></span>
<span class="p"><</span><span class="nt">tr</span><span class="p">></span>
<span class="p"><</span><span class="nt">th</span><span class="p">></span>KEY<span class="p"></</span><span class="nt">th</span><span class="p">></span>
<span class="p"><</span><span class="nt">th</span><span class="p">></span>VALUE<span class="p"></</span><span class="nt">th</span><span class="p">></span>
<span class="p"><</span><span class="nt">th</span><span class="p">></</span><span class="nt">th</span><span class="p">></span>
<span class="p"></</span><span class="nt">tr</span><span class="p">></span>
<span class="p"></</span><span class="nt">thead</span><span class="p">></span>
<span class="p"><</span><span class="nt">tbody</span><span class="p">></span>
<span class="p"><</span><span class="nt">tr</span> <span class="err">*</span><span class="na">ngFor</span><span class="o">=</span><span class="s">"let pair of pairs; let idx = index …</span></code></pre></div><p>I will show how to progressively add new rows to the table as the last row is being edited in <a href="https://angular.io">Angular</a></p>
<p>In, <code>app.component.html</code></p>
<div class="highlight"><pre><span></span><code><span class="p"><</span><span class="nt">table</span> <span class="na">class</span><span class="o">=</span><span class="s">"progressive-table"</span><span class="p">></span>
<span class="p"><</span><span class="nt">thead</span><span class="p">></span>
<span class="p"><</span><span class="nt">tr</span><span class="p">></span>
<span class="p"><</span><span class="nt">th</span><span class="p">></span>KEY<span class="p"></</span><span class="nt">th</span><span class="p">></span>
<span class="p"><</span><span class="nt">th</span><span class="p">></span>VALUE<span class="p"></</span><span class="nt">th</span><span class="p">></span>
<span class="p"><</span><span class="nt">th</span><span class="p">></</span><span class="nt">th</span><span class="p">></span>
<span class="p"></</span><span class="nt">tr</span><span class="p">></span>
<span class="p"></</span><span class="nt">thead</span><span class="p">></span>
<span class="p"><</span><span class="nt">tbody</span><span class="p">></span>
<span class="p"><</span><span class="nt">tr</span> <span class="err">*</span><span class="na">ngFor</span><span class="o">=</span><span class="s">"let pair of pairs; let idx = index"</span><span class="p">></span>
<span class="p"><</span><span class="nt">td</span><span class="p">></span>
<span class="p"><</span><span class="nt">input</span> <span class="na">type</span><span class="o">=</span><span class="s">"text"</span> <span class="err">[(</span><span class="na">ngModel</span><span class="err">)]="</span><span class="na">pair</span><span class="err">.</span><span class="na">Key</span><span class="err">"</span> <span class="err">(</span><span class="na">ngModelChange</span><span class="err">)="</span><span class="na">pairValueChanged</span><span class="err">(</span><span class="na">idx</span><span class="err">)"</span><span class="p">></span>
<span class="p"></</span><span class="nt">td</span><span class="p">></span>
<span class="p"><</span><span class="nt">td</span><span class="p">></span>
<span class="p"><</span><span class="nt">input</span> <span class="na">type</span><span class="o">=</span><span class="s">"text"</span> <span class="err">[(</span><span class="na">ngModel</span><span class="err">)]="</span><span class="na">pair</span><span class="err">.</span><span class="na">Value</span><span class="err">"</span> <span class="err">(</span><span class="na">ngModelChange</span><span class="err">)="</span><span class="na">pairValueChanged</span><span class="err">(</span><span class="na">idx</span><span class="err">)"</span><span class="p">></span>
<span class="p"></</span><span class="nt">td</span><span class="p">></span>
<span class="p"><</span><span class="nt">td</span><span class="p">></span>
<span class="p"><</span><span class="nt">i</span> <span class="na">class</span><span class="o">=</span><span class="s">"fa fa-times"</span> <span class="na">aria-hidden</span><span class="o">=</span><span class="s">"true"</span> <span class="err">(</span><span class="na">click</span><span class="err">)="</span><span class="na">deletePair</span><span class="err">(</span><span class="na">idx</span><span class="err">)"</span><span class="p">></</span><span class="nt">i</span><span class="p">></span>
<span class="p"></</span><span class="nt">td</span><span class="p">></span>
<span class="p"></</span><span class="nt">tr</span><span class="p">></span>
<span class="p"></</span><span class="nt">tbody</span><span class="p">></span>
<span class="p"></</span><span class="nt">table</span><span class="p">></span>
</code></pre></div>
<p>In, <code>app.component.scss</code></p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="nc">progressive-table</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">font-size</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">width</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">%</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">border-collapse</span><span class="p">:</span><span class="w"> </span><span class="kc">collapse</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="err">tr</span><span class="w"> </span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="k">height</span><span class="p">:</span><span class="w"> </span><span class="mi">30</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nt">th</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">background-color</span><span class="p">:</span><span class="w"> </span><span class="mh">#eee</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">padding-left</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nt">th</span><span class="p">:</span><span class="nd">last-child</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">width</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nt">td</span><span class="o">,</span><span class="w"> </span><span class="nt">th</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">border</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="kt">px</span><span class="w"> </span><span class="kc">solid</span><span class="w"> </span><span class="kc">lightgray</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">text-align</span><span class="p">:</span><span class="w"> </span><span class="kc">left</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nt">input</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">width</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="kt">%</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">padding</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="kt">px</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">border</span><span class="p">:</span><span class="w"> </span><span class="kc">none</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">box-sizing</span><span class="p">:</span><span class="w"> </span><span class="kc">border-box</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kp">-moz-</span><span class="k">box-sizing</span><span class="p">:</span><span class="w"> </span><span class="kc">border-box</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kp">-webkit-</span><span class="k">box-sizing</span><span class="p">:</span><span class="w"> </span><span class="kc">border-box</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nt">i</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">cursor</span><span class="p">:</span><span class="w"> </span><span class="kc">pointer</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="err">}</span><span class="w"></span>
</code></pre></div>
<p>And finally in, <code>app.component.ts</code></p>
<div class="highlight"><pre><span></span><code><span class="kd">const</span><span class="w"> </span><span class="nx">emptyPair</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="s1">'Key'</span><span class="o">:</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="s1">'Value'</span><span class="o">:</span><span class="w"> </span><span class="s1">''</span><span class="p">};</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="nx">pairs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[{...</span><span class="k">this</span><span class="p">.</span><span class="nx">emptyPair</span><span class="p">}];</span><span class="w"></span>
<span class="nx">pairValueChanged</span><span class="p">(</span><span class="nx">idx</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">idx</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">pairs</span><span class="p">.</span><span class="nx">length</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mf">1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">pairs</span><span class="p">.</span><span class="nx">push</span><span class="p">({...</span><span class="k">this</span><span class="p">.</span><span class="nx">emptyPair</span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="nx">deletePair</span><span class="p">(</span><span class="nx">idx</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">pairs</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">idx</span><span class="p">,</span><span class="w"> </span><span class="mf">1</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">pairs</span><span class="p">.</span><span class="nx">length</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="mf">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">pairs</span><span class="p">.</span><span class="nx">push</span><span class="p">({...</span><span class="k">this</span><span class="p">.</span><span class="nx">emptyPair</span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Final output looks like this;
<img alt="Output" src="/images/progressive-table.png"></p>
<p>Hope this helps!</p>Gambler's Ruin problem simulation2019-12-01T07:00:00+05:302019-12-01T07:00:00+05:30ArunDhaJtag:www.arundhaj.com,2019-12-01:/blog/gamblers-ruin-problem-simulation.html<p>Detailed problem statement can be found here <a href="https://en.wikipedia.org/wiki/Gambler%27s_ruin">Gambler's Ruin</a></p>
<p>General solution to the problem is;</p>
<p>$1- a_1 = a_1 \sum\limits_{i = 1}^{k-1} \left(\frac{1 - p}{p}\right)^i$</p>
<p>For a fair game of $p = \frac{1}{2}$, $a_1 = \frac{1}{k}$</p>
<p>and, for an unfair game of $p …</p><p>Detailed problem statement can be found here <a href="https://en.wikipedia.org/wiki/Gambler%27s_ruin">Gambler's Ruin</a></p>
<p>General solution to the problem is;</p>
<p>$1- a_1 = a_1 \sum\limits_{i = 1}^{k-1} \left(\frac{1 - p}{p}\right)^i$</p>
<p>For a fair game of $p = \frac{1}{2}$, $a_1 = \frac{1}{k}$</p>
<p>and, for an unfair game of $p \neq \frac{1}{2}$</p>
<p>$a_1 =\frac{\left(\frac{1-p}{p}\right)-1}{\left(\frac{1-p}{p}\right)^k-1}$</p>
<p>Below is the python implementation of the simulation of a fair game of probability $\frac{1}{2}$, with player 1 balance of $98 & player 2 balance of $2</p>
<div class="highlight"><pre><span></span><code><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="n">number_of_simulations</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="c1"># Create an array of wins of length equal to number_of_simulations</span>
<span class="n">wins</span> <span class="o">=</span> <span class="p">[</span><span class="kc">False</span><span class="p">]</span> <span class="o">*</span> <span class="n">number_of_simulations</span>
<span class="c1"># Fair Game</span>
<span class="n">prob</span> <span class="o">=</span> <span class="mf">0.5</span>
<span class="n">p1_balance</span> <span class="o">=</span> <span class="mi">98</span>
<span class="n">p2_balance</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">number_of_simulations</span><span class="p">):</span>
<span class="n">p1b</span> <span class="o">=</span> <span class="n">p1_balance</span>
<span class="n">p2b</span> <span class="o">=</span> <span class="n">p2_balance</span>
<span class="c1"># Play until p1 or p2 is getting ruined</span>
<span class="k">while</span> <span class="n">p1b</span> <span class="o">></span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">p2b</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
<span class="n">p1_win</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o"><</span> <span class="n">prob</span>
<span class="n">p1b</span> <span class="o">=</span> <span class="n">p1b</span> <span class="o">+</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">p1_win</span> <span class="k">else</span> <span class="n">p1b</span> <span class="o">-</span> <span class="mi">1</span>
<span class="n">p2b</span> <span class="o">=</span> <span class="n">p2b</span> <span class="o">-</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">p1_win</span> <span class="k">else</span> <span class="n">p2b</span> <span class="o">+</span> <span class="mi">1</span>
<span class="c1"># Set to True, if p1 still not ruined </span>
<span class="n">wins</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">p1b</span> <span class="o">></span> <span class="mi">0</span>
<span class="n">prob_p1_wins</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">wins</span><span class="p">)</span><span class="o">/</span><span class="n">number_of_simulations</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Player 1</span><span class="se">\'</span><span class="s1">s probability of wining is: </span><span class="si">{</span><span class="n">prob_p1_wins</span><span class="si">}</span><span class="s1">'</span><span class="p">)</span>
</code></pre></div>
<p>Output is;</p>
<div class="highlight"><pre><span></span><code>Player <span class="m">1</span><span class="err">'</span>s probability of wining is: <span class="m">0</span>.979
</code></pre></div>
<p>On repeatedly running the above simulation for the player 1 balance ranging from 1 to 99, we will see that probability increases as the initial amount increases;</p>
<p><img alt="Probability of winning" src="/images/prob_of_winning.png"></p>Python f-string2019-10-17T14:00:00+05:302019-10-17T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2019-10-17:/blog/python-f-string.html<p>f-strings is a short for "formatted strings", provides a convenient way to embed expressions inside string literals. This supports more formatting options than the old <code>str.format()</code> method supports.</p>
<p>Few examples;</p>
<p>String operation</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Arunkumar'</span>
<span class="o">>>></span> <span class="sa">f</span><span class="s1">'My name is </span><span class="si">{</span><span class="n">name</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span><span class="si">}</span><span class="s1">'</span>
<span class="s1">'My name is ARUNKUMAR'</span>
</code></pre></div>
<p>Float formatting</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">math</span>
<span class="o">>>></span> <span class="sa">f …</span></code></pre></div><p>f-strings is a short for "formatted strings", provides a convenient way to embed expressions inside string literals. This supports more formatting options than the old <code>str.format()</code> method supports.</p>
<p>Few examples;</p>
<p>String operation</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Arunkumar'</span>
<span class="o">>>></span> <span class="sa">f</span><span class="s1">'My name is </span><span class="si">{</span><span class="n">name</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span><span class="si">}</span><span class="s1">'</span>
<span class="s1">'My name is ARUNKUMAR'</span>
</code></pre></div>
<p>Float formatting</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">math</span>
<span class="o">>>></span> <span class="sa">f</span><span class="s1">'math.pi:.5f'</span>
<span class="s1">'3.14159'</span>
</code></pre></div>
<p>Datetime formatting</p>
<div class="highlight"><pre><span></span><code><span class="n">this_day</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="sa">f</span><span class="s1">'Yesteday was </span><span class="si">{</span><span class="n">this_day</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span><span class="si">:</span><span class="s1">%A</span><span class="si">}</span><span class="s1"> and tommorow is </span><span class="si">{</span><span class="n">this_day</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span><span class="si">:</span><span class="s1">%A</span><span class="si">}</span><span class="s1">'</span>
<span class="s1">'Yesterday was Wednesday and tommorow is Friday'</span>
</code></pre></div>
<p>f-string token looks like;</p>
<div class="highlight"><pre><span></span><code>f <span class="s1">' <text> { <expression> <optional !s, !r, or !a> <optional : format specifier> } <text> ... '</span>
</code></pre></div>
<p>Starting from <code>3.8</code> f-string supports "=" symbol for documenting expressions</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="nb">sum</span> <span class="o">=</span> <span class="mi">5</span> <span class="o">+</span> <span class="mi">6</span>
<span class="o">>>></span> <span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="nb">sum</span><span class="si">=}</span><span class="s1">'</span>
<span class="s1">'sum=11'</span>
</code></pre></div>
<p>Hope this helps!</p>Setting content-type for files uploaded to S32019-02-21T11:00:00+05:302019-02-21T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2019-02-21:/blog/setting-content-type-files-s3.html<p>While uploading files to AWS S3 using Python <a href="https://boto3.readthedocs.io">Boto3</a> library, it would by default set the <code>content-type</code> as binary. Below snippet would set the appropriate content type based on the file extension.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">mimetypes</span>
<span class="o">...</span>
<span class="n">file_mime_type</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">mimetypes</span><span class="o">.</span><span class="n">guess_type</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="o">...</span>
<span class="n">bucket_obj</span><span class="o">.</span><span class="n">upload_file</span><span class="p">(</span><span class="n">local_path</span><span class="p">,</span> <span class="n">s3_path</span><span class="p">,</span> <span class="n">ExtraArgs</span><span class="o">=</span><span class="p">{</span><span class="s1">'ContentType'</span><span class="p">:</span> <span class="n">file_mime_type</span><span class="p">})</span>
<span class="o">...</span>
</code></pre></div>
<p>The above snippet would …</p><p>While uploading files to AWS S3 using Python <a href="https://boto3.readthedocs.io">Boto3</a> library, it would by default set the <code>content-type</code> as binary. Below snippet would set the appropriate content type based on the file extension.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">mimetypes</span>
<span class="o">...</span>
<span class="n">file_mime_type</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">mimetypes</span><span class="o">.</span><span class="n">guess_type</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="o">...</span>
<span class="n">bucket_obj</span><span class="o">.</span><span class="n">upload_file</span><span class="p">(</span><span class="n">local_path</span><span class="p">,</span> <span class="n">s3_path</span><span class="p">,</span> <span class="n">ExtraArgs</span><span class="o">=</span><span class="p">{</span><span class="s1">'ContentType'</span><span class="p">:</span> <span class="n">file_mime_type</span><span class="p">})</span>
<span class="o">...</span>
</code></pre></div>
<p>The above snippet would predict most of the common mime types. In case if some mime types are missing, it could be fixed by explicitly adding the type. Put the below snippet before calling <code>guess_type</code> method.</p>
<div class="highlight"><pre><span></span><code><span class="n">mimetypes</span><span class="o">.</span><span class="n">add_type</span><span class="p">(</span><span class="s1">'application/json'</span><span class="p">,</span> <span class="s1">'.json'</span><span class="p">)</span>
</code></pre></div>
<p>Hope this helps!</p>Syncing files to AWS S3 bucket using AWS CLI2019-01-10T11:00:00+05:302019-01-10T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2019-01-10:/blog/syncing-files-aws-s3-bucket-awscli.html<p><a href="https://docs.aws.amazon.com/cli/latest/index.html">AWS CLI</a> is a command line tool used for interacting with AWS services.</p>
<p>In this post, I'll share the command to sync your local files to S3 bucket by removing files in the bucket that are not present in the local folder.</p>
<div class="highlight"><pre><span></span><code>$ aws s3 sync . s3://bucket-name --delete
</code></pre></div>
<p><code>--delete</code> option …</p><p><a href="https://docs.aws.amazon.com/cli/latest/index.html">AWS CLI</a> is a command line tool used for interacting with AWS services.</p>
<p>In this post, I'll share the command to sync your local files to S3 bucket by removing files in the bucket that are not present in the local folder.</p>
<div class="highlight"><pre><span></span><code>$ aws s3 sync . s3://bucket-name --delete
</code></pre></div>
<p><code>--delete</code> option deletes the files in bucket that doesn't exists in the local folder.</p>
<p>Hope this helps!</p>Securing your webapp with AWS Cognito2019-01-03T10:30:00+05:302019-01-03T10:30:00+05:30ArunDhaJtag:www.arundhaj.com,2019-01-03:/blog/securing-webapp-with-aws-cognito.html<p><a href="https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html">AWS Cognito</a> provides authentication, authorization, and user management for your webapps. In this article I will show <a href="https://angular.io/">Angular</a> snippets to perform authentication with AWS Cognito credentials.</p>
<p>First, I'll show the <code>CognitoService</code> class with just <code>signIn</code> functionality. I've removed other operations like register, confirm registration, etc. from the snippet for simplicity …</p><p><a href="https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html">AWS Cognito</a> provides authentication, authorization, and user management for your webapps. In this article I will show <a href="https://angular.io/">Angular</a> snippets to perform authentication with AWS Cognito credentials.</p>
<p>First, I'll show the <code>CognitoService</code> class with just <code>signIn</code> functionality. I've removed other operations like register, confirm registration, etc. from the snippet for simplicity.</p>
<div class="highlight"><pre><span></span><code><span class="kd">const</span><span class="w"> </span><span class="nx">poolData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">UserPoolId</span><span class="o">:</span><span class="w"> </span><span class="s1">'us-east-1_XXXXXXXXX'</span><span class="p">,</span><span class="w"> </span><span class="c1">// Your user pool id here</span><span class="w"></span>
<span class="w"> </span><span class="nx">ClientId</span><span class="o">:</span><span class="w"> </span><span class="s1">'XXXXXXXXXXXXXXXXXXXXXXXXXX'</span><span class="w"> </span><span class="c1">// Your client id here</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="kd">const</span><span class="w"> </span><span class="nx">userPool</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">CognitoUserPool</span><span class="p">(</span><span class="nx">poolData</span><span class="p">);</span><span class="w"></span>
<span class="kd">@Injectable</span><span class="p">({</span><span class="w"></span>
<span class="w"> </span><span class="nx">providedIn</span><span class="o">:</span><span class="w"> </span><span class="s1">'root'</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
<span class="k">export</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nx">CognitoService</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kr">constructor</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">signIn</span><span class="p">(</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="nx">password</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">authenticationData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Username</span><span class="o">:</span><span class="w"> </span><span class="kt">username</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Password</span><span class="o">:</span><span class="w"> </span><span class="kt">password</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">authenticationDetails</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">AuthenticationDetails</span><span class="p">(</span><span class="nx">authenticationData</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">userData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Username</span><span class="o">:</span><span class="w"> </span><span class="kt">username</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Pool</span><span class="o">:</span><span class="w"> </span><span class="kt">userPool</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">cognitoUser</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">CognitoUser</span><span class="p">(</span><span class="nx">userData</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">Observable</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="nx">observer</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">cognitoUser</span><span class="p">.</span><span class="nx">authenticateUser</span><span class="p">(</span><span class="nx">authenticationDetails</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">onSuccess</span><span class="o">:</span><span class="w"> </span><span class="p">(</span><span class="nx">result</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">observer</span><span class="p">.</span><span class="nx">next</span><span class="p">(</span><span class="nx">result</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="nx">observer</span><span class="p">.</span><span class="nx">complete</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nx">onFailure</span><span class="o">:</span><span class="w"> </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="nx">observer</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>And in the <code>LoginComponent</code> class</p>
<div class="highlight"><pre><span></span><code><span class="k">export</span><span class="w"> </span><span class="kd">class</span><span class="w"> </span><span class="nx">LoginComponent</span><span class="w"> </span><span class="k">implements</span><span class="w"> </span><span class="nx">OnInit</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">userCredentialsModel</span><span class="o">:</span><span class="w"> </span><span class="kt">UserCredentialsModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{};</span><span class="w"></span>
<span class="w"> </span><span class="nx">loading</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="nx">errorMessage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kr">constructor</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">private</span><span class="w"> </span><span class="nx">router</span><span class="o">:</span><span class="w"> </span><span class="kt">Router</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">private</span><span class="w"> </span><span class="nx">cognitoService</span><span class="o">:</span><span class="w"> </span><span class="kt">CognitoService</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">login</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">errorMessage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">loading</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">true</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">cognitoService</span><span class="p">.</span><span class="nx">signIn</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">userCredentialsModel</span><span class="p">.</span><span class="nx">userName</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">userCredentialsModel</span><span class="p">.</span><span class="nx">password</span><span class="p">).</span><span class="nx">subscribe</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">loading</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">router</span><span class="p">.</span><span class="nx">navigate</span><span class="p">([</span><span class="s1">'/securedpage'</span><span class="p">]);</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="nx">error</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">loading</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">false</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">errorMessage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">error</span><span class="p">.</span><span class="nx">message</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Hope this helps!</p>Serverless application architecture in Python with AWS Lambda2017-10-25T15:00:00+05:302017-10-25T15:00:00+05:30ArunDhaJtag:www.arundhaj.com,2017-10-25:/blog/serverless-application-architecture-python-aws-lambda.html<p><a href="http://docs.aws.amazon.com/lambda/latest/dg/welcome.html">AWS Lambda</a> is a compute service that lets you run code without provisioning or managing servers. In this article I will show how to create and manage simple AWS Lambda function written in <a href="https://www.python.org">Python</a>.</p>
<p>Create files and folders as shown below. Technically, <code>app.py</code> is sufficient to create a function …</p><p><a href="http://docs.aws.amazon.com/lambda/latest/dg/welcome.html">AWS Lambda</a> is a compute service that lets you run code without provisioning or managing servers. In this article I will show how to create and manage simple AWS Lambda function written in <a href="https://www.python.org">Python</a>.</p>
<p>Create files and folders as shown below. Technically, <code>app.py</code> is sufficient to create a function, however the following structure lets you to separate the concerns into its specified layers.</p>
<div class="highlight"><pre><span></span><code>py3lambda
├───mylib
│ ├───__init__.py
│ ├───helper.py
│ ├───model.py
│ └───service.py
├───packages
├───app.py
├───config.json
└───manage.py
</code></pre></div>
<p><code>mylib</code> folder shall contain your application files.</p>
<p><code>package</code> folder should contain any 3rd party packages as required by your application. To install a package to this directory use the following command.</p>
<div class="highlight"><pre><span></span><code>$ pip install pymysql -t .<span class="se">\p</span>ackages
</code></pre></div>
<p><code>app.py</code> contains the main entry point of AWS Lambda function.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">mylib</span> <span class="kn">import</span> <span class="n">helper</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span>
<span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="sd">""" Lamdba entry point. """</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inside lambda_handler1"</span><span class="p">)</span>
<span class="n">helper</span><span class="o">.</span><span class="n">some_helper</span><span class="p">()</span>
<span class="k">return</span> <span class="p">{</span><span class="s1">'Status'</span><span class="p">:</span> <span class="s1">'Success'</span><span class="p">}</span>
</code></pre></div>
<p>Add configurations in <code>config.json</code> file, that goes to AWS Lambda environment variables.</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"<<VARIABLE_NAME1>>"</span><span class="p">:</span><span class="w"> </span><span class="s2">"<<variable_value1>>"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"<<VARIABLE_NAME2>>"</span><span class="p">:</span><span class="w"> </span><span class="s2">"<<variable_value2>>"</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>And finally, the script to manage AWS Lambda function.
<code>manage.py</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">io</span>
<span class="kn">import</span> <span class="nn">zipfile</span>
<span class="kn">import</span> <span class="nn">boto3</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="k">def</span> <span class="nf">archive</span><span class="p">():</span>
<span class="n">buf</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">BytesIO</span><span class="p">()</span>
<span class="k">with</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZIP_DEFLATED</span><span class="p">)</span> <span class="k">as</span> <span class="n">zfh</span><span class="p">:</span>
<span class="c1"># Add my main file</span>
<span class="n">zfh</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'app.py'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">root</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">files</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="s1">'mylib'</span><span class="p">):</span>
<span class="k">for</span> <span class="n">file</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
<span class="n">zfh</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">file</span><span class="p">))</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="s1">'packages'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">root</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">files</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="s1">'.'</span><span class="p">):</span>
<span class="k">for</span> <span class="n">file</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
<span class="n">zfh</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">file</span><span class="p">))</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="s1">'..'</span><span class="p">)</span>
<span class="n">buf</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">pkg</span> <span class="o">=</span> <span class="n">buf</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">return</span> <span class="n">pkg</span>
<span class="k">def</span> <span class="nf">environment</span><span class="p">():</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'config.json'</span><span class="p">)</span> <span class="k">as</span> <span class="n">config_file</span><span class="p">:</span>
<span class="n">env_json</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">config_file</span><span class="p">)</span>
<span class="k">return</span> <span class="n">env_json</span>
<span class="k">def</span> <span class="nf">zip_create_function</span><span class="p">():</span>
<span class="n">lambda_client</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s1">'lambda'</span><span class="p">)</span>
<span class="n">pkg</span> <span class="o">=</span> <span class="n">archive</span><span class="p">()</span>
<span class="n">env_json</span> <span class="o">=</span> <span class="n">environment</span><span class="p">()</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">lambda_client</span><span class="o">.</span><span class="n">create_function</span><span class="p">(</span>
<span class="n">FunctionName</span><span class="o">=</span><span class="s1">'Py3Lambda'</span><span class="p">,</span>
<span class="n">Runtime</span><span class="o">=</span><span class="s1">'python3.6'</span><span class="p">,</span>
<span class="n">Role</span><span class="o">=</span><span class="s1">'<<your role ARN>>'</span><span class="p">,</span>
<span class="n">Handler</span><span class="o">=</span><span class="s1">'app.lambda_handler'</span><span class="p">,</span>
<span class="n">Code</span><span class="o">=</span><span class="p">{</span><span class="s1">'ZipFile'</span><span class="p">:</span> <span class="n">pkg</span><span class="p">},</span>
<span class="n">Environment</span><span class="o">=</span><span class="p">{</span>
<span class="s1">'Variables'</span><span class="p">:</span> <span class="n">env_json</span>
<span class="p">},</span>
<span class="n">Description</span><span class="o">=</span><span class="s1">'Python3 Lambda function'</span><span class="p">,</span>
<span class="n">Timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">,</span>
<span class="n">MemorySize</span><span class="o">=</span><span class="mi">128</span><span class="p">,</span>
<span class="n">Publish</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">response</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">zip_update_function</span><span class="p">():</span>
<span class="n">lambda_client</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s1">'lambda'</span><span class="p">)</span>
<span class="n">pkg</span> <span class="o">=</span> <span class="n">archive</span><span class="p">()</span>
<span class="n">env_json</span> <span class="o">=</span> <span class="n">environment</span><span class="p">()</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">lambda_client</span><span class="o">.</span><span class="n">update_function_code</span><span class="p">(</span>
<span class="n">FunctionName</span><span class="o">=</span><span class="s1">'Py3Lambda'</span><span class="p">,</span>
<span class="n">ZipFile</span><span class="o">=</span><span class="n">pkg</span><span class="p">,</span>
<span class="n">Publish</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">response</span><span class="p">))</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">lambda_client</span><span class="o">.</span><span class="n">update_function_configuration</span><span class="p">(</span>
<span class="n">FunctionName</span><span class="o">=</span><span class="s1">'Py3Lambda'</span><span class="p">,</span>
<span class="n">Environment</span><span class="o">=</span><span class="p">{</span>
<span class="s1">'Variables'</span><span class="p">:</span> <span class="n">env_json</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">response</span><span class="p">))</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">2</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Invalid option'</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'create'</span><span class="p">:</span>
<span class="n">zip_create_function</span><span class="p">()</span>
<span class="k">elif</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'update'</span><span class="p">:</span>
<span class="n">zip_update_function</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Invalid option'</span><span class="p">)</span>
</code></pre></div>
<p>To run the above script</p>
<div class="highlight"><pre><span></span><code>$ python manage.py create<span class="p">|</span>update
</code></pre></div>
<p>Hope this helps!</p>Merging an upstream repository into your fork with TortoiseGit2017-10-05T14:00:00+05:302017-10-05T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2017-10-05:/blog/merging-upstream-repository-into-fork-tortoise-git.html<p>To update your <a href="https://github.com">GitHub</a> forked repository to that of the remote using <a href="https://tortoisegit.org">TortoiseGit</a>, follow the below listed steps:</p>
<ul>
<li>Add <code>upstream</code> repository that has to be synced with your fork</li>
</ul>
<p><code>TortoiseGit</code> -> <code>Settings</code> -> <code>Remote</code></p>
<p><img alt="Add Upstream" src="/images/fork-merge1.png"></p>
<ul>
<li>Fetch the <code>upstream</code> repository</li>
</ul>
<p><code>TortoiseGit</code> -> <code>Fetch</code></p>
<p><img alt="Fetch" src="/images/fork-merge2.png"></p>
<ul>
<li>Pull the <code>upstream</code> repository</li>
</ul>
<p><code>TortoiseGit</code> -> <code>Pull</code></p>
<p><img alt="Pull" src="/images/fork-merge3.png"></p>
<ul>
<li>
<p>Resolve any merge conflicts</p>
</li>
<li>
<p>Push to …</p></li></ul><p>To update your <a href="https://github.com">GitHub</a> forked repository to that of the remote using <a href="https://tortoisegit.org">TortoiseGit</a>, follow the below listed steps:</p>
<ul>
<li>Add <code>upstream</code> repository that has to be synced with your fork</li>
</ul>
<p><code>TortoiseGit</code> -> <code>Settings</code> -> <code>Remote</code></p>
<p><img alt="Add Upstream" src="/images/fork-merge1.png"></p>
<ul>
<li>Fetch the <code>upstream</code> repository</li>
</ul>
<p><code>TortoiseGit</code> -> <code>Fetch</code></p>
<p><img alt="Fetch" src="/images/fork-merge2.png"></p>
<ul>
<li>Pull the <code>upstream</code> repository</li>
</ul>
<p><code>TortoiseGit</code> -> <code>Pull</code></p>
<p><img alt="Pull" src="/images/fork-merge3.png"></p>
<ul>
<li>
<p>Resolve any merge conflicts</p>
</li>
<li>
<p>Push to the <code>origin master</code> branch</p>
</li>
</ul>
<p><code>TortoiseGit</code> -> <code>Push</code></p>
<p><img alt="Push" src="/images/fork-merge4.png"></p>
<p>Hope this helps!</p>Read and Write DataFrame from Database using PySpark2017-03-20T14:00:00+05:302017-03-20T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2017-03-20:/blog/read-write-dataframe-from-database-pyspark.html<p>To load a DataFrame from a MySQL table in <a href="http://spark.apache.org/docs/latest/api/python">PySpark</a></p>
<div class="highlight"><pre><span></span><code><span class="nv">source_df</span> <span class="o">=</span> sqlContext.read.format<span class="o">(</span><span class="s1">'jdbc'</span><span class="o">)</span>.options<span class="o">(</span>
<span class="nv">url</span><span class="o">=</span><span class="s1">'jdbc:mysql://localhost/database_name'</span>,
<span class="nv">driver</span><span class="o">=</span><span class="s1">'com.mysql.jdbc.Driver'</span>,
<span class="nv">dbtable</span><span class="o">=</span><span class="s1">'SourceTableName'</span>,
<span class="nv">user</span><span class="o">=</span><span class="s1">'your_user_name'</span>,
<span class="nv">password</span><span class="o">=</span><span class="s1">'your_password'</span><span class="o">)</span>.load<span class="o">()</span>
</code></pre></div>
<p>And to write a DataFrame to a MySQL table</p>
<div class="highlight"><pre><span></span><code>destination_df.write.format<span class="o">(</span><span class="s1">'jdbc'</span><span class="o">)</span>.options<span class="o">(</span>
<span class="nv">url</span><span class="o">=</span><span class="s1">'jdbc:mysql …</span></code></pre></div><p>To load a DataFrame from a MySQL table in <a href="http://spark.apache.org/docs/latest/api/python">PySpark</a></p>
<div class="highlight"><pre><span></span><code><span class="nv">source_df</span> <span class="o">=</span> sqlContext.read.format<span class="o">(</span><span class="s1">'jdbc'</span><span class="o">)</span>.options<span class="o">(</span>
<span class="nv">url</span><span class="o">=</span><span class="s1">'jdbc:mysql://localhost/database_name'</span>,
<span class="nv">driver</span><span class="o">=</span><span class="s1">'com.mysql.jdbc.Driver'</span>,
<span class="nv">dbtable</span><span class="o">=</span><span class="s1">'SourceTableName'</span>,
<span class="nv">user</span><span class="o">=</span><span class="s1">'your_user_name'</span>,
<span class="nv">password</span><span class="o">=</span><span class="s1">'your_password'</span><span class="o">)</span>.load<span class="o">()</span>
</code></pre></div>
<p>And to write a DataFrame to a MySQL table</p>
<div class="highlight"><pre><span></span><code>destination_df.write.format<span class="o">(</span><span class="s1">'jdbc'</span><span class="o">)</span>.options<span class="o">(</span>
<span class="nv">url</span><span class="o">=</span><span class="s1">'jdbc:mysql://localhost/database_name'</span>,
<span class="nv">driver</span><span class="o">=</span><span class="s1">'com.mysql.jdbc.Driver'</span>,
<span class="nv">dbtable</span><span class="o">=</span><span class="s1">'DestinationTableName'</span>,
<span class="nv">user</span><span class="o">=</span><span class="s1">'your_user_name'</span>,
<span class="nv">password</span><span class="o">=</span><span class="s1">'your_password'</span><span class="o">)</span>.mode<span class="o">(</span><span class="s1">'append'</span><span class="o">)</span>.save<span class="o">()</span>
</code></pre></div>
<p>While submitting the <a href="http://spark.apache.org">spark</a> program, use the following command</p>
<div class="highlight"><pre><span></span><code>bin/spark-submit --jars external/mysql-connector-java-5.1.40-bin.jar
/path_to_your_program/spark_database.py
</code></pre></div>
<p>Hope this helps!</p>Calculate difference with previous row in PySpark2017-03-15T11:00:00+05:302017-03-15T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2017-03-15:/blog/calculate-difference-with-previous-row-in-pyspark.html<p>To find the difference between the current row value and the previous row value in <a href="http://spark.apache.org">spark</a> programming with <a href="http://spark.apache.org/docs/latest/api/python">PySpark</a> is as below</p>
<p>Let say, we have the following DataFrame and we shall now calculate the difference of values between consecutive rows.</p>
<div class="highlight"><pre><span></span><code>+---+-----+
<span class="p">|</span> id<span class="p">|</span>value<span class="p">|</span>
+---+-----+
<span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span>
<span class="p">|</span> <span class="m">2</span><span class="p">|</span> <span class="m">66</span><span class="p">|</span>
<span class="p">|</span> <span class="m">3</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span>
<span class="p">|</span> <span class="m">4 …</span></code></pre></div><p>To find the difference between the current row value and the previous row value in <a href="http://spark.apache.org">spark</a> programming with <a href="http://spark.apache.org/docs/latest/api/python">PySpark</a> is as below</p>
<p>Let say, we have the following DataFrame and we shall now calculate the difference of values between consecutive rows.</p>
<div class="highlight"><pre><span></span><code>+---+-----+
<span class="p">|</span> id<span class="p">|</span>value<span class="p">|</span>
+---+-----+
<span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span>
<span class="p">|</span> <span class="m">2</span><span class="p">|</span> <span class="m">66</span><span class="p">|</span>
<span class="p">|</span> <span class="m">3</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span>
<span class="p">|</span> <span class="m">4</span><span class="p">|</span> <span class="m">68</span><span class="p">|</span>
<span class="p">|</span> <span class="m">5</span><span class="p">|</span> <span class="m">71</span><span class="p">|</span>
+---+-----+
</code></pre></div>
<p>We first, create a new column with previous row's value as below</p>
<div class="highlight"><pre><span></span><code>+---+-----+----------+
<span class="p">|</span> id<span class="p">|</span>value<span class="p">|</span>prev_value<span class="p">|</span>
+---+-----+----------+
<span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span> null<span class="p">|</span>
<span class="p">|</span> <span class="m">2</span><span class="p">|</span> <span class="m">66</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span>
<span class="p">|</span> <span class="m">3</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span> <span class="m">66</span><span class="p">|</span>
<span class="p">|</span> <span class="m">4</span><span class="p">|</span> <span class="m">68</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span>
<span class="p">|</span> <span class="m">5</span><span class="p">|</span> <span class="m">71</span><span class="p">|</span> <span class="m">68</span><span class="p">|</span>
+---+-----+----------+
</code></pre></div>
<p>The following code snippet finds us the desired results.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">pyspark</span> <span class="kn">import</span> <span class="n">SparkContext</span>
<span class="kn">from</span> <span class="nn">pyspark.sql</span> <span class="kn">import</span> <span class="n">SQLContext</span>
<span class="kn">from</span> <span class="nn">pyspark.sql</span> <span class="kn">import</span> <span class="n">functions</span> <span class="k">as</span> <span class="n">F</span>
<span class="kn">from</span> <span class="nn">pyspark.sql.window</span> <span class="kn">import</span> <span class="n">Window</span>
<span class="n">sc</span> <span class="o">=</span> <span class="n">SparkContext</span><span class="p">(</span><span class="n">appName</span><span class="o">=</span><span class="s2">"PrevRowDiffApp"</span><span class="p">)</span>
<span class="n">sqlc</span> <span class="o">=</span> <span class="n">SQLContext</span><span class="p">(</span><span class="n">sc</span><span class="p">)</span>
<span class="n">rdd</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">65</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">66</span><span class="p">),</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">65</span><span class="p">),</span> <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">68</span><span class="p">),</span> <span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">71</span><span class="p">)])</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">sqlc</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">(</span><span class="n">rdd</span><span class="p">,</span> <span class="p">[</span><span class="s2">"id"</span><span class="p">,</span> <span class="s2">"value"</span><span class="p">])</span>
<span class="n">my_window</span> <span class="o">=</span> <span class="n">Window</span><span class="o">.</span><span class="n">partitionBy</span><span class="p">()</span><span class="o">.</span><span class="n">orderBy</span><span class="p">(</span><span class="s2">"id"</span><span class="p">)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">withColumn</span><span class="p">(</span><span class="s2">"prev_value"</span><span class="p">,</span> <span class="n">F</span><span class="o">.</span><span class="n">lag</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">value</span><span class="p">)</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">my_window</span><span class="p">))</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">withColumn</span><span class="p">(</span><span class="s2">"diff"</span><span class="p">,</span> <span class="n">F</span><span class="o">.</span><span class="n">when</span><span class="p">(</span><span class="n">F</span><span class="o">.</span><span class="n">isnull</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">value</span> <span class="o">-</span> <span class="n">df</span><span class="o">.</span><span class="n">prev_value</span><span class="p">),</span> <span class="mi">0</span><span class="p">)</span>
<span class="o">.</span><span class="n">otherwise</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">value</span> <span class="o">-</span> <span class="n">df</span><span class="o">.</span><span class="n">prev_value</span><span class="p">))</span>
<span class="n">df</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</code></pre></div>
<p>The final result is in diff column.</p>
<div class="highlight"><pre><span></span><code>+---+-----+----------+----+
<span class="p">|</span> id<span class="p">|</span>value<span class="p">|</span>prev_value<span class="p">|</span>diff<span class="p">|</span>
+---+-----+----------+----+
<span class="p">|</span> <span class="m">1</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span> null<span class="p">|</span> <span class="m">0</span><span class="p">|</span>
<span class="p">|</span> <span class="m">2</span><span class="p">|</span> <span class="m">66</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span> <span class="m">1</span><span class="p">|</span>
<span class="p">|</span> <span class="m">3</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span> <span class="m">66</span><span class="p">|</span> -1<span class="p">|</span>
<span class="p">|</span> <span class="m">4</span><span class="p">|</span> <span class="m">68</span><span class="p">|</span> <span class="m">65</span><span class="p">|</span> <span class="m">3</span><span class="p">|</span>
<span class="p">|</span> <span class="m">5</span><span class="p">|</span> <span class="m">71</span><span class="p">|</span> <span class="m">68</span><span class="p">|</span> <span class="m">3</span><span class="p">|</span>
+---+-----+----------+----+
</code></pre></div>
<p>Hope this helps!</p>Getting started with AWS Kinesis using Python2016-12-21T14:00:00+05:302016-12-21T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2016-12-21:/blog/getting-started-kinesis-python.html<p>Amazon Kinesis is a fully managed stream hosted on AWS. It is used to collect and process large streams of data in real time. Along with Kinesis Analytics, Kinesis Firehose, AWS Lambda, AWS S3, AWS EMR you can build a robust distributed application to power your real-time monitoring dashboards, do …</p><p>Amazon Kinesis is a fully managed stream hosted on AWS. It is used to collect and process large streams of data in real time. Along with Kinesis Analytics, Kinesis Firehose, AWS Lambda, AWS S3, AWS EMR you can build a robust distributed application to power your real-time monitoring dashboards, do massive scale batch analytics, etc.</p>
<p>First create a Kinesis stream using the following <code>aws-cli</code> command</p>
<div class="highlight"><pre><span></span><code>> aws kinesis create-stream --stream-name python-stream --shard-count <span class="m">1</span>
</code></pre></div>
<p>The following code, say <code>kinesis_producer.py</code> will put records to the stream continuosly every 5 seconds</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">boto3</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="kn">import</span> <span class="nn">calendar</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="n">my_stream_name</span> <span class="o">=</span> <span class="s1">'python-stream'</span>
<span class="n">kinesis_client</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s1">'kinesis'</span><span class="p">,</span> <span class="n">region_name</span><span class="o">=</span><span class="s1">'us-east-1'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">put_to_stream</span><span class="p">(</span><span class="n">thing_id</span><span class="p">,</span> <span class="n">property_value</span><span class="p">,</span> <span class="n">property_timestamp</span><span class="p">):</span>
<span class="n">payload</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'prop'</span><span class="p">:</span> <span class="nb">str</span><span class="p">(</span><span class="n">property_value</span><span class="p">),</span>
<span class="s1">'timestamp'</span><span class="p">:</span> <span class="nb">str</span><span class="p">(</span><span class="n">property_timestamp</span><span class="p">),</span>
<span class="s1">'thing_id'</span><span class="p">:</span> <span class="n">thing_id</span>
<span class="p">}</span>
<span class="nb">print</span> <span class="n">payload</span>
<span class="n">put_response</span> <span class="o">=</span> <span class="n">kinesis_client</span><span class="o">.</span><span class="n">put_record</span><span class="p">(</span>
<span class="n">StreamName</span><span class="o">=</span><span class="n">my_stream_name</span><span class="p">,</span>
<span class="n">Data</span><span class="o">=</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">payload</span><span class="p">),</span>
<span class="n">PartitionKey</span><span class="o">=</span><span class="n">thing_id</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">property_value</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">120</span><span class="p">)</span>
<span class="n">property_timestamp</span> <span class="o">=</span> <span class="n">calendar</span><span class="o">.</span><span class="n">timegm</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span><span class="o">.</span><span class="n">timetuple</span><span class="p">())</span>
<span class="n">thing_id</span> <span class="o">=</span> <span class="s1">'aa-bb'</span>
<span class="n">put_to_stream</span><span class="p">(</span><span class="n">thing_id</span><span class="p">,</span> <span class="n">property_value</span><span class="p">,</span> <span class="n">property_timestamp</span><span class="p">)</span>
<span class="c1"># wait for 5 second</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
</code></pre></div>
<p>Start consuming with <code>kinesis_consumer.py</code> as shown below. The following consumer will start consuming the data as the producer puts to the stream.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">boto3</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="n">my_stream_name</span> <span class="o">=</span> <span class="s1">'python-stream'</span>
<span class="n">kinesis_client</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s1">'kinesis'</span><span class="p">,</span> <span class="n">region_name</span><span class="o">=</span><span class="s1">'us-east-1'</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">kinesis_client</span><span class="o">.</span><span class="n">describe_stream</span><span class="p">(</span><span class="n">StreamName</span><span class="o">=</span><span class="n">my_stream_name</span><span class="p">)</span>
<span class="n">my_shard_id</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="s1">'StreamDescription'</span><span class="p">][</span><span class="s1">'Shards'</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s1">'ShardId'</span><span class="p">]</span>
<span class="n">shard_iterator</span> <span class="o">=</span> <span class="n">kinesis_client</span><span class="o">.</span><span class="n">get_shard_iterator</span><span class="p">(</span><span class="n">StreamName</span><span class="o">=</span><span class="n">my_stream_name</span><span class="p">,</span>
<span class="n">ShardId</span><span class="o">=</span><span class="n">my_shard_id</span><span class="p">,</span>
<span class="n">ShardIteratorType</span><span class="o">=</span><span class="s1">'LATEST'</span><span class="p">)</span>
<span class="n">my_shard_iterator</span> <span class="o">=</span> <span class="n">shard_iterator</span><span class="p">[</span><span class="s1">'ShardIterator'</span><span class="p">]</span>
<span class="n">record_response</span> <span class="o">=</span> <span class="n">kinesis_client</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="n">ShardIterator</span><span class="o">=</span><span class="n">my_shard_iterator</span><span class="p">,</span>
<span class="n">Limit</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="k">while</span> <span class="s1">'NextShardIterator'</span> <span class="ow">in</span> <span class="n">record_response</span><span class="p">:</span>
<span class="n">record_response</span> <span class="o">=</span> <span class="n">kinesis_client</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="n">ShardIterator</span><span class="o">=</span><span class="n">record_response</span><span class="p">[</span><span class="s1">'NextShardIterator'</span><span class="p">],</span>
<span class="n">Limit</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="nb">print</span> <span class="n">record_response</span>
<span class="c1"># wait for 5 seconds</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
</code></pre></div>
<p>Assuming you have the credentials appropriately configured.</p>
<p>Hope this helps!</p>AWS DynamoDB full table scan2016-12-16T11:00:00+05:302016-12-16T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2016-12-16:/blog/aws-dynamodb-full-table-scan.html<p>Amazon DynamoDB is a NoSQL database service hosted on AWS. It is a fully managed and scalable document store database. It is quite similar to MongoDB.</p>
<p>As the data grows, scan operation on full table would return parts of the data with the <code>LastEvaluatedKey</code>. The application should initiate scan again …</p><p>Amazon DynamoDB is a NoSQL database service hosted on AWS. It is a fully managed and scalable document store database. It is quite similar to MongoDB.</p>
<p>As the data grows, scan operation on full table would return parts of the data with the <code>LastEvaluatedKey</code>. The application should initiate scan again from the <code>LastEvaluatedKey</code>. Below code snippet shows how to do it.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">boto3</span>
<span class="n">dynamodb_client</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">(</span><span class="s1">'dynamodb'</span><span class="p">,</span> <span class="n">region_name</span><span class="o">=</span><span class="s1">'us-east-1'</span><span class="p">)</span>
<span class="n">my_table</span> <span class="o">=</span> <span class="n">dynamodb_client</span><span class="o">.</span><span class="n">Table</span><span class="p">(</span><span class="s1">'MyTableName'</span><span class="p">)</span>
<span class="n">filter_expression</span> <span class="o">=</span> <span class="n">Key</span><span class="p">(</span><span class="s1">'id'</span><span class="p">)</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">record_id</span><span class="p">)</span> <span class="o">&</span>
<span class="n">Key</span><span class="p">(</span><span class="s1">'timestamp'</span><span class="p">)</span><span class="o">.</span><span class="n">between</span><span class="p">(</span><span class="n">long</span><span class="p">(</span><span class="n">start_date</span><span class="p">),</span> <span class="n">long</span><span class="p">(</span><span class="n">end_date</span><span class="p">))</span>
<span class="n">result_item</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">result_data</span> <span class="o">=</span> <span class="n">my_table</span><span class="o">.</span><span class="n">scan</span><span class="p">(</span>
<span class="n">FilterExpression</span><span class="o">=</span><span class="n">filter_expression</span>
<span class="p">)</span>
<span class="n">result_item</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">result_data</span><span class="p">[</span><span class="s1">'Items'</span><span class="p">])</span>
<span class="k">while</span> <span class="s1">'LastEvaluatedKey'</span> <span class="ow">in</span> <span class="n">result_data</span><span class="p">:</span>
<span class="n">result_data</span> <span class="o">=</span> <span class="n">my_table</span><span class="o">.</span><span class="n">scan</span><span class="p">(</span>
<span class="n">FilterExpression</span><span class="o">=</span><span class="n">filter_expression</span><span class="p">,</span>
<span class="n">ExclusiveStartKey</span><span class="o">=</span><span class="n">result_data</span><span class="p">[</span><span class="s1">'LastEvaluatedKey'</span><span class="p">]</span>
<span class="p">)</span>
<span class="n">result_item</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">result_data</span><span class="p">[</span><span class="s1">'Items'</span><span class="p">])</span>
</code></pre></div>
<p>Assuming you have the credentials appropriately configured.</p>
<p>Hope this helps!</p>Manage multiple SSH keys for GIT2016-11-29T14:00:00+05:302016-11-29T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2016-11-29:/blog/manage-multiple-ssh-keys-for-git.html<p>Configuring and managing multiple SSH keys for GIT accounts, in Ubuntu is shown below:</p>
<p>Using <code>ssh-keygen</code> create a public/private key and name it as <code>user1_github</code>, <code>user1_bitbucket</code></p>
<p>create a file <code>~/.ssh/config</code> and put the following in it.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># user1 github account</span>
Host user1_github
HostName github.com
PreferredAuthentications publickey
IdentityFile ~/.ssh …</code></pre></div><p>Configuring and managing multiple SSH keys for GIT accounts, in Ubuntu is shown below:</p>
<p>Using <code>ssh-keygen</code> create a public/private key and name it as <code>user1_github</code>, <code>user1_bitbucket</code></p>
<p>create a file <code>~/.ssh/config</code> and put the following in it.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># user1 github account</span>
Host user1_github
HostName github.com
PreferredAuthentications publickey
IdentityFile ~/.ssh/user1_github
<span class="c1"># user1 bitbucket account</span>
Host user1_bitbucket
HostName bitbucket.org
PreferredAuthentications publickey
IdentityFile ~/.ssh/user1_bitbucket
</code></pre></div>
<p>Once the configuration is done, you can checkout as below</p>
<div class="highlight"><pre><span></span><code> $ git clone git@user1_github:user1/my_github_repo.git
$ git clone git@user1_bitbucket:user1/my_bitbucket_repo.git
</code></pre></div>
<p>Hope this helps!</p>Resample timeseries data with custom function2016-10-20T11:00:00+05:302016-10-20T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2016-10-20:/blog/resample-timeseries-with-custom-function.html<p>With timeseries data we often require to resample on different intervel to feed in to our analytics model.</p>
<p>Pandas resample have a built-in list of widely used methods. However, if the built-in methods are not sufficient, it is always possible to write a custom function to resample.</p>
<p>This post shows …</p><p>With timeseries data we often require to resample on different intervel to feed in to our analytics model.</p>
<p>Pandas resample have a built-in list of widely used methods. However, if the built-in methods are not sufficient, it is always possible to write a custom function to resample.</p>
<p>This post shows an example.</p>
<p>Say, we have a months temperature data captured every hour. We shall calculate the number of times in a day the temperature crossed 40 degree celsius.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">seaborn</span> <span class="k">as</span> <span class="nn">sns</span>
<span class="n">ts_index</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">date_range</span><span class="p">(</span><span class="n">start</span><span class="o">=</span><span class="s1">'10/01/2016'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">'10/30/2016'</span><span class="p">,</span> <span class="n">freq</span><span class="o">=</span><span class="s1">'H'</span><span class="p">)</span>
<span class="n">temperature_series</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">60</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">ts_index</span><span class="p">)),</span> <span class="n">index</span><span class="o">=</span><span class="n">ts_index</span><span class="p">)</span>
<span class="n">temperature_series_mean</span> <span class="o">=</span> <span class="n">temperature_series</span>
<span class="o">.</span><span class="n">resample</span><span class="p">(</span><span class="s1">'D'</span><span class="p">)</span>
<span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="n">temperature_series_gt_40</span> <span class="o">=</span> <span class="n">temperature_series</span>
<span class="o">.</span><span class="n">resample</span><span class="p">(</span><span class="s1">'D'</span><span class="p">)</span>
<span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">day_array</span><span class="p">:</span> <span class="nb">sum</span><span class="p">(</span><span class="n">day_array</span> <span class="o">></span> <span class="mi">40</span><span class="p">))</span>
<span class="n">fig</span><span class="p">,</span> <span class="n">ax1</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
<span class="n">ax2</span> <span class="o">=</span> <span class="n">ax1</span><span class="o">.</span><span class="n">twinx</span><span class="p">()</span>
<span class="c1"># Day's mean shown as line chart</span>
<span class="n">plt_mean</span> <span class="o">=</span> <span class="n">ax1</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">temperature_series_mean</span><span class="p">,</span> <span class="s1">'c-'</span><span class="p">)</span>
<span class="c1"># Day's count shown as dot chart</span>
<span class="n">plt_gt40</span> <span class="o">=</span> <span class="n">ax2</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">temperature_series_gt_40</span><span class="p">,</span> <span class="s1">'bo'</span><span class="p">)</span>
<span class="n">ax1</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">'Date'</span><span class="p">)</span>
<span class="n">ax1</span><span class="o">.</span><span class="n">set_ylabel</span><span class="p">(</span><span class="s1">'Day</span><span class="se">\'</span><span class="s1">s mean temerature'</span><span class="p">)</span>
<span class="n">ax2</span><span class="o">.</span><span class="n">set_ylabel</span><span class="p">(</span><span class="s1">'Number of times temperature crossed 40'</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">(</span><span class="n">handles</span><span class="o">=</span><span class="n">plt_mean</span><span class="p">,</span> <span class="n">loc</span><span class="o">=</span><span class="s1">'upper left'</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">rcParams</span><span class="p">[</span><span class="s1">'figure.figsize'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">13</span><span class="p">,</span><span class="mi">7</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</code></pre></div>
<p><img alt="Resample Timeseries" src="/images/resample-timeseries.png"></p>Birthday paradox2015-06-25T14:00:00+05:302015-06-25T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2015-06-25:/blog/birthday-paradox.html<p>Building python visualization of the famous <a href="https://en.wikipedia.org/wiki/Birthday_problem">Birthday paradox or Birthday problem</a>.</p>
<p>Birthday problem, is to find the probability of a pair from the given set of randomly chosen people to have same birthday.</p>
<p>Given, we have 365 days (ignoring February 29th). The following chart shows how the probability increases with …</p><p>Building python visualization of the famous <a href="https://en.wikipedia.org/wiki/Birthday_problem">Birthday paradox or Birthday problem</a>.</p>
<p>Birthday problem, is to find the probability of a pair from the given set of randomly chosen people to have same birthday.</p>
<p>Given, we have 365 days (ignoring February 29th). The following chart shows how the probability increases with the number of people.</p>
<p><img alt="Birthday paradox" src="/images/birthday-paradox.png"></p>
<p>The interesting observation is that, with only 23 people we already have 50% probability to have same birthday. It reaches 99.9% with just 70 people.</p>
<p>Python code used for constructing the visualization</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">math</span>
<span class="kn">from</span> <span class="nn">decimal</span> <span class="kn">import</span> <span class="n">Decimal</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">seaborn</span> <span class="k">as</span> <span class="nn">sns</span>
<span class="c1"># Function to find the probability</span>
<span class="k">def</span> <span class="nf">sameday_prob</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="n">pBar</span> <span class="o">=</span> <span class="n">Decimal</span><span class="p">(</span><span class="n">math</span><span class="o">.</span><span class="n">factorial</span><span class="p">(</span><span class="mi">365</span><span class="p">))</span><span class="o">/</span><span class="p">(</span><span class="nb">pow</span><span class="p">(</span><span class="mi">365</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span> <span class="o">*</span> <span class="n">math</span><span class="o">.</span><span class="n">factorial</span><span class="p">(</span><span class="mi">365</span> <span class="o">-</span> <span class="n">n</span><span class="p">))</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">pBar</span>
<span class="c1"># Number of peoples</span>
<span class="n">x</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">120</span><span class="p">)</span>
<span class="c1"># find probabilities for each set of people</span>
<span class="n">y</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">yBar</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">persons</span> <span class="ow">in</span> <span class="n">x</span><span class="p">:</span>
<span class="n">prob</span> <span class="o">=</span> <span class="n">sameday_prob</span><span class="p">(</span><span class="n">persons</span><span class="p">)</span>
<span class="n">y</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">prob</span><span class="p">)</span>
<span class="n">yBar</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">prob</span><span class="p">)</span>
<span class="c1"># Plot the chart</span>
<span class="n">line1</span><span class="p">,</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">"Matching"</span><span class="p">)</span>
<span class="n">line2</span><span class="p">,</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">yBar</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">"No matching"</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">vlines</span><span class="p">(</span><span class="mi">23</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">sameday_prob</span><span class="p">(</span><span class="mi">23</span><span class="p">),</span> <span class="n">linestyles</span><span class="o">=</span><span class="sa">u</span><span class="s1">'dashed'</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">hlines</span><span class="p">(</span><span class="n">sameday_prob</span><span class="p">(</span><span class="mi">23</span><span class="p">),</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="n">linestyles</span><span class="o">=</span><span class="sa">u</span><span class="s1">'dashed'</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">(</span><span class="n">handles</span><span class="o">=</span><span class="p">[</span><span class="n">line1</span><span class="p">,</span> <span class="n">line2</span><span class="p">],</span> <span class="n">loc</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">'Number of people'</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">'Probability of match'</span><span class="p">)</span>
</code></pre></div>
<p>The calculation used to find the probability is</p>
<p>$$\bar P(n) = \frac{365!}{365^n (365 - n)!}$$
$$P(n) = 1 - \bar P(n)$$</p>Adding empty directory within GIT2015-06-16T15:00:00+05:302015-06-16T15:00:00+05:30ArunDhaJtag:www.arundhaj.com,2015-06-16:/blog/empty-directory-within-git.html<p>While maintaining GIT repositories, in some cases, we might be required to add an empty directory as a placeholder to mean something or we wanted all the files in a directory to stay local. However in GIT we can add only files to the repository and not the directory. </p>
<p>I …</p><p>While maintaining GIT repositories, in some cases, we might be required to add an empty directory as a placeholder to mean something or we wanted all the files in a directory to stay local. However in GIT we can add only files to the repository and not the directory. </p>
<p>I came across the following <a href="http://stackoverflow.com/questions/115983/how-can-i-add-an-empty-directory-to-a-git-repository">stackoverflow</a> post, which suggest a workaround.</p>
<p>It basically suggest to add a <code>.gitignore</code> file in the directory with the following line.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Ignore everything in this directory</span>
*
<span class="c1"># Except this file</span>
!.gitignore
</code></pre></div>
<p>Hope this helps.</p>Tornado Error during WebSocket handshake2015-05-09T22:30:00+05:302015-05-09T22:30:00+05:30ArunDhaJtag:www.arundhaj.com,2015-05-09:/blog/tornado-error-during-websocket-handshake.html<p>I was getting following exeption in WebSocket client, when trying to connect <a href="http://tornado.readthedocs.org/en/latest/websocket.html">Tornado</a> WebSocket server.</p>
<div class="highlight"><pre><span></span><code>WebSocket connection to <span class="s1">'ws://localhost:5678/echo'</span> failed: Error during WebSocket handshake: Unexpected response code: <span class="m">403</span>
</code></pre></div>
<p>and on the server-side log</p>
<div class="highlight"><pre><span></span><code>WARNING:tornado.access:403 GET /echo <span class="o">(</span>::1<span class="o">)</span> <span class="m">6</span>.00ms
</code></pre></div>
<p>A simple Echo WebSocket server …</p><p>I was getting following exeption in WebSocket client, when trying to connect <a href="http://tornado.readthedocs.org/en/latest/websocket.html">Tornado</a> WebSocket server.</p>
<div class="highlight"><pre><span></span><code>WebSocket connection to <span class="s1">'ws://localhost:5678/echo'</span> failed: Error during WebSocket handshake: Unexpected response code: <span class="m">403</span>
</code></pre></div>
<p>and on the server-side log</p>
<div class="highlight"><pre><span></span><code>WARNING:tornado.access:403 GET /echo <span class="o">(</span>::1<span class="o">)</span> <span class="m">6</span>.00ms
</code></pre></div>
<p>A simple Echo WebSocket server code.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">tornado</span> <span class="kn">import</span> <span class="n">websocket</span><span class="p">,</span> <span class="n">ioloop</span>
<span class="k">class</span> <span class="nc">EchoWebSocket</span><span class="p">(</span><span class="n">websocket</span><span class="o">.</span><span class="n">WebSocketHandler</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">open</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"WebSocket opened"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">write_message</span><span class="p">(</span><span class="sa">u</span><span class="s2">"You said: "</span> <span class="o">+</span> <span class="n">message</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"WebSocket closed"</span><span class="p">)</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">web</span><span class="o">.</span><span class="n">Application</span><span class="p">([</span>
<span class="p">(</span><span class="sa">r</span><span class="s1">'/echo'</span><span class="p">,</span> <span class="n">EchoWebSocket</span><span class="p">),</span>
<span class="p">])</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">listen</span><span class="p">(</span><span class="mi">5678</span><span class="p">)</span>
<span class="n">ioloop</span><span class="o">.</span><span class="n">IOLoop</span><span class="o">.</span><span class="n">instance</span><span class="p">()</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</code></pre></div>
<p>It is because the server denies cross-orgin request. We should overide <code>check_origin</code> method to allow the clients to connect to the server.</p>
<div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">check_origin</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">origin</span><span class="p">):</span>
<span class="k">return</span> <span class="kc">True</span>
</code></pre></div>
<p>The above snippet would allow all origin. However, to allow certain set of domains/sub-domains, appropriate validation has to be implemented.</p>
<p>Hope this helps.</p>Binding port 80 to tomcat in Ubuntu2015-04-27T22:30:00+05:302015-04-27T22:30:00+05:30ArunDhaJtag:www.arundhaj.com,2015-04-27:/blog/binding-port-80-tomcat-ubuntu.html<p>Edit the file <code>/etc/default/tomcat7</code></p>
<p>change the line </p>
<div class="highlight"><pre><span></span><code><span class="c1">#AUTHBIND=no</span>
</code></pre></div>
<p>to</p>
<div class="highlight"><pre><span></span><code><span class="nv">AUTHBIND</span><span class="o">=</span>yes
</code></pre></div>
<p>and run the following commands</p>
<div class="highlight"><pre><span></span><code>$ sudo touch /etc/authbind/byport/80
$ sudo chmod <span class="m">500</span> /etc/authbind/byport/80
$ sudo chown tomcat7 /etc/authbind/byport/80
</code></pre></div>
<p>Hope this helps.</p>Scikit-learn DecisionTree with categorical data2015-03-22T18:00:00+05:302015-03-22T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2015-03-22:/blog/scikit-decision-tree-with-categorical-data.html<p>In this post, I'll walk through scikit-learn's <a href="http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier">DecisionTreeClassifier</a> from loading the data, fitting the model and prediction.</p>
<p>I'm going to use the <a href="https://github.com/arundhaj/datamining_class/blob/master/data/vertebrate.csv">vertebrate</a> dataset from the book <a href="http://www-users.cs.umn.edu/~kumar/dmbook/index.php">Introduction to Data Mining</a> by Tan, Steinbach and Kumar. </p>
<p>We need to predict the class label of the last record from the dataset …</p><p>In this post, I'll walk through scikit-learn's <a href="http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier">DecisionTreeClassifier</a> from loading the data, fitting the model and prediction.</p>
<p>I'm going to use the <a href="https://github.com/arundhaj/datamining_class/blob/master/data/vertebrate.csv">vertebrate</a> dataset from the book <a href="http://www-users.cs.umn.edu/~kumar/dmbook/index.php">Introduction to Data Mining</a> by Tan, Steinbach and Kumar. </p>
<p>We need to predict the class label of the last record from the dataset.</p>
<p>As you see the data is categorical. We need to vectorize the features so that, we can feed to the classifier. And it is done as follows.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">sklearn.feature_extraction</span> <span class="kn">import</span> <span class="n">DictVectorizer</span>
<span class="n">X_dict</span> <span class="o">=</span> <span class="n">X_feature</span><span class="o">.</span><span class="n">T</span><span class="o">.</span><span class="n">to_dict</span><span class="p">()</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
<span class="n">vect</span> <span class="o">=</span> <span class="n">DictVectorizer</span><span class="p">(</span><span class="n">sparse</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">X_vector</span> <span class="o">=</span> <span class="n">vect</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X_dict</span><span class="p">)</span>
</code></pre></div>
<p>On converting the pandas DataFrame <code>X_feature</code> to python dictionary <code>X_dict</code>. </p>
<p><code>X_dict</code> looks like;</p>
<div class="highlight"><pre><span></span><code><span class="p">[{</span><span class="s1">'Aerial Creature'</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span>
<span class="s1">'Aquatic Creature'</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span>
<span class="s1">'Body Temperature'</span><span class="p">:</span> <span class="s1">'warm-blooded'</span><span class="p">,</span>
<span class="s1">'Gives Birth'</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span>
<span class="s1">'Has Legs'</span><span class="p">:</span> <span class="s1">'yes'</span><span class="p">,</span>
<span class="s1">'Hibernates'</span><span class="p">:</span> <span class="s1">'no'</span><span class="p">,</span>
<span class="s1">'Skin Cover'</span><span class="p">:</span> <span class="s1">'hair'</span><span class="p">},</span>
<span class="o">...</span>
<span class="p">]</span>
</code></pre></div>
<p>Then to vector <code>X_vector</code></p>
<div class="highlight"><pre><span></span><code><span class="o">[[</span> <span class="m">1</span>., <span class="m">0</span>., <span class="m">1</span>., <span class="m">0</span>., <span class="m">0</span>., <span class="m">0</span>., <span class="m">1</span>., <span class="m">0</span>., <span class="m">1</span>., <span class="m">0</span>., <span class="m">1</span>., <span class="m">1</span>., <span class="m">0</span>.,
<span class="m">0</span>., <span class="m">0</span>., <span class="m">1</span>., <span class="m">0</span>., <span class="m">0</span>., <span class="m">0</span>.<span class="o">]</span>,
...
<span class="o">]</span>
</code></pre></div>
<p>The vectorizer matrix contains features in the following format.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span><span class="s1">'Aerial Creature=no'</span>,
<span class="s1">'Aerial Creature=yes'</span>,
<span class="s1">'Aquatic Creature=no'</span>,
<span class="s1">'Aquatic Creature=semi'</span>,
<span class="s1">'Aquatic Creature=yes'</span>,
<span class="s1">'Body Temperature=cold-blooded'</span>,
<span class="s1">'Body Temperature=warm-blooded'</span>,
<span class="s1">'Gives Birth=no'</span>,
<span class="s1">'Gives Birth=yes'</span>,
<span class="s1">'Has Legs=no'</span>,
<span class="s1">'Has Legs=yes'</span>,
<span class="s1">'Hibernates=no'</span>,
<span class="s1">'Hibernates=yes'</span>,
<span class="s1">'Skin Cover=feathers'</span>,
<span class="s1">'Skin Cover=fur'</span>,
<span class="s1">'Skin Cover=hair'</span>,
<span class="s1">'Skin Cover=none'</span>,
<span class="s1">'Skin Cover=quills'</span>,
<span class="s1">'Skin Cover=scales'</span><span class="o">]</span>
</code></pre></div>
<p>For vectorizing the class labels</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">LabelEncoder</span>
<span class="n">le</span> <span class="o">=</span> <span class="n">LabelEncoder</span><span class="p">()</span>
<span class="n">y_train</span> <span class="o">=</span> <span class="n">le</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s1">'Class Label'</span><span class="p">][:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
</code></pre></div>
<p>With the X and y vectorized, we can now use the <code>DecisionTreeClassifier</code> for fitting the model and to do prediction.</p>
<p>Find the ipython notebook <a href="http://nbviewer.ipython.org/github/arundhaj/datamining_class/blob/master/decision_tree.ipynb">here</a></p>Building Hadoop source code2014-11-25T23:00:00+05:302014-11-25T23:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-11-25:/blog/building-hadoop-source-code.html<p>The <a href="http://hadoop.apache.org">Apache Hadoop</a> is a framework that allows for distributed processing of large data sets across clusters of computers using MapReduce. </p>
<p>The steps listed below is to build and package hadoop from source code. This guide assumes a fresh installation of Ubuntu 14.04 version.</p>
<ol>
<li>Let's start with installing <a href="http://www.oracle.com/technetwork/java/javase/downloads/index.html">Oracle …</a></li></ol><p>The <a href="http://hadoop.apache.org">Apache Hadoop</a> is a framework that allows for distributed processing of large data sets across clusters of computers using MapReduce. </p>
<p>The steps listed below is to build and package hadoop from source code. This guide assumes a fresh installation of Ubuntu 14.04 version.</p>
<ol>
<li>Let's start with installing <a href="http://www.oracle.com/technetwork/java/javase/downloads/index.html">Oracle Java</a>. First navigate to home folder <code>cd</code></li>
</ol>
<div class="highlight"><pre><span></span><code>mkdir installations <span class="o">&&</span> <span class="nb">cd</span> installations
wget --no-check-certificate --no-cookies --header <span class="s2">"Cookie:oraclelicense=accept-securebackup-cookie"</span> http://download.oracle.com/otn-pub/java/jdk/7u71-b14/jdk-7u71-linux-x64.tar.gz
sudo mkdir /usr/lib/jvm
sudo tar xvzf jdk-7u71-linux-x64.tar.gz -C /usr/lib/jvm/
</code></pre></div>
<ul>
<li>Then we shall install <a href="http://maven.apache.org">Apache Maven</a>. Hadoop's main build system.</li>
</ul>
<div class="highlight"><pre><span></span><code>wget http://apache.tradebit.com/pub/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz
sudo tar xvzf apache-maven-3.2.3-bin.tar.gz - C /usr/local/
</code></pre></div>
<ul>
<li>Now, set the environmental variables by appending the following lines at the end of <code>~/.bashrc</code> file.</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="nb">export</span> <span class="nv">JAVA_HOME</span><span class="o">=</span>/usr/lib/jvm/jdk1.7.0_71
<span class="nb">export</span> <span class="nv">M2_HOME</span><span class="o">=</span>/usr/local/apache-maven-3.2.3
<span class="nb">export</span> <span class="nv">PATH</span><span class="o">=</span><span class="nv">$PATH</span>:<span class="nv">$JAVA_HOME</span>/bin:<span class="nv">$M2_HOME</span>/bin
</code></pre></div>
<ul>
<li>Lets check if java and maven being installed correcly.</li>
</ul>
<div class="highlight"><pre><span></span><code>java -version
mvn --version
</code></pre></div>
<ul>
<li>Install the following dependent packages</li>
</ul>
<div class="highlight"><pre><span></span><code>sudo apt-get update
sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev rsync openssh-server
</code></pre></div>
<ul>
<li>Setup ssh for password-less login</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span>
ssh-keygen -t rsa -P <span class="s2">""</span>
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
</code></pre></div>
<ul>
<li>Check if you are able to ssh without password. And make sure to exit and return to main shell.</li>
</ul>
<div class="highlight"><pre><span></span><code>ssh localhost
</code></pre></div>
<ul>
<li>Install <a href="https://developers.google.com/protocol-buffers">protobuf</a> v2.5.0. This is the highest version Hadoop supports.</li>
</ul>
<div class="highlight"><pre><span></span><code>mkdir source_code <span class="o">&&</span> <span class="nb">cd</span> source_code
wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar xvzf protobuf-2.5.0.tar.gz
<span class="nb">cd</span> protobuf-2.5.0.tar.gz
./configure
make
make check
sudo make install
sudo ldconfig
protoc --version
</code></pre></div>
<ul>
<li>Get the Hadoop source code. You can clone the repository from the source control. For this tutorial, I've downloaded the source tar.</li>
</ul>
<div class="highlight"><pre><span></span><code>wget http://apache.claz.org/hadoop/common/hadoop-2.5.2/hadoop-2.5.2-src.tar.gz
tar xvzf hadoop-2.5.2-src.tar.gz
<span class="nb">cd</span> hadoop-2.5.2-src
mvn package -Pdist -Dtar -DskipTests
</code></pre></div>
<p>Depending upon the internet speed, the compile may take 10 minutes to 1 hour. Once the build is successful, the distribution would be available in <code>hadoop-dist/target/hadoop-2.5.2.tar.gz</code></p>
<p>Hope this helps.</p>Pig script to process CSV file with quotes and multiline2014-09-25T11:00:00+05:302014-09-25T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-09-25:/blog/pig-script-process-csv-with-quotes-multiline.html<p>While writing <a href="http://pig.apache.org">Pig</a> script, usually we use <code>PigStorage</code> for loading a CSV file.</p>
<p>Consider a sample CSV file in the following format.</p>
<div class="highlight"><pre><span></span><code><span class="mi">2</span><span class="p">,</span><span class="n">Loading</span> <span class="n">successfull</span><span class="p">,</span><span class="mi">2014</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">25</span>
<span class="mi">3</span><span class="p">,</span><span class="n">Loading</span> <span class="n">successfull</span><span class="p">,</span><span class="mi">2014</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">25</span>
<span class="mi">4</span><span class="p">,</span><span class="n">Loading</span> <span class="n">successfull</span><span class="p">,</span><span class="mi">2014</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">25</span>
</code></pre></div>
<p>can be loaded as</p>
<div class="highlight"><pre><span></span><code><span class="n">logs</span> <span class="o">=</span> <span class="n">LOAD</span> <span class="s1">'log_folder/log_file.csv'</span> <span class="n">USING …</span></code></pre></div><p>While writing <a href="http://pig.apache.org">Pig</a> script, usually we use <code>PigStorage</code> for loading a CSV file.</p>
<p>Consider a sample CSV file in the following format.</p>
<div class="highlight"><pre><span></span><code><span class="mi">2</span><span class="p">,</span><span class="n">Loading</span> <span class="n">successfull</span><span class="p">,</span><span class="mi">2014</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">25</span>
<span class="mi">3</span><span class="p">,</span><span class="n">Loading</span> <span class="n">successfull</span><span class="p">,</span><span class="mi">2014</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">25</span>
<span class="mi">4</span><span class="p">,</span><span class="n">Loading</span> <span class="n">successfull</span><span class="p">,</span><span class="mi">2014</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">25</span>
</code></pre></div>
<p>can be loaded as</p>
<div class="highlight"><pre><span></span><code><span class="n">logs</span> <span class="o">=</span> <span class="n">LOAD</span> <span class="s1">'log_folder/log_file.csv'</span> <span class="n">USING</span> <span class="n">PigStorage</span><span class="p">(</span><span class="s1">','</span><span class="p">)</span> <span class="n">AS</span> <span class="p">(</span><span class="nb">id</span><span class="p">:</span> <span class="n">long</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">chararray</span><span class="p">,</span> <span class="n">timestamp</span><span class="p">:</span> <span class="n">chararray</span><span class="p">);</span>
</code></pre></div>
<p>However, I had a CSV file containing double quotes and also a single record spanning multiple lines. In the format shown below.</p>
<div class="highlight"><pre><span></span><code><span class="s2">"2"</span>,<span class="s2">"Loading</span>
<span class="s2">successfull"</span>,<span class="s2">"2014-09-25"</span>
<span class="s2">"3"</span>,<span class="s2">"Loading</span>
<span class="s2">successfull"</span>,<span class="s2">"2014-09-25"</span>
<span class="s2">"4"</span>,<span class="s2">"Loading</span>
<span class="s2">successfull"</span>,<span class="s2">"2014-09-25"</span>
</code></pre></div>
<p>this sort of CSV file can be loaded using <code>org.apache.pig.piggybank.storage.CSVExcelStorage</code></p>
<div class="highlight"><pre><span></span><code><span class="n">logs</span> <span class="o">=</span> <span class="n">LOAD</span> <span class="s1">'log_folder/log_file.csv'</span> <span class="n">USING</span> <span class="n">org</span><span class="o">.</span><span class="n">apache</span><span class="o">.</span><span class="n">pig</span><span class="o">.</span><span class="n">piggybank</span><span class="o">.</span><span class="n">storage</span><span class="o">.</span><span class="n">CSVExcelStorage</span><span class="p">(</span><span class="s1">','</span><span class="p">,</span> <span class="s1">'YES_MULTILINE'</span><span class="p">)</span> <span class="n">AS</span> <span class="p">(</span><span class="nb">id</span><span class="p">:</span> <span class="n">long</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">chararray</span><span class="p">,</span> <span class="n">timestamp</span><span class="p">:</span> <span class="n">chararray</span><span class="p">);</span>
</code></pre></div>
<p>Hope this helps.</p>Split array based on difference with NumPy2014-09-05T11:00:00+05:302014-09-05T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-09-05:/blog/split-array-on-difference-numpy.html<p>I had a <a href="http://www.numpy.org">NumPy</a> array of numbers, which I had to split based on the change of value.</p>
<p>For example, consider an array as shown below.</p>
<div class="highlight"><pre><span></span><code><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="mf">112.0</span><span class="p">,</span> <span class="mf">111.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span> <span class="mf">111.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span>
<span class="mf">113.0</span><span class="p">,</span> <span class="mf">114.0</span><span class="p">,</span> <span class="mf">114 …</span></code></pre></div><p>I had a <a href="http://www.numpy.org">NumPy</a> array of numbers, which I had to split based on the change of value.</p>
<p>For example, consider an array as shown below.</p>
<div class="highlight"><pre><span></span><code><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="mf">112.0</span><span class="p">,</span> <span class="mf">111.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span> <span class="mf">111.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span>
<span class="mf">113.0</span><span class="p">,</span> <span class="mf">114.0</span><span class="p">,</span> <span class="mf">114.0</span><span class="p">,</span> <span class="mf">115.0</span><span class="p">,</span> <span class="mf">117.0</span><span class="p">,</span> <span class="mf">102.0</span><span class="p">,</span> <span class="mf">102.0</span><span class="p">,</span> <span class="mf">102.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span>
<span class="mf">103.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">]</span>
</code></pre></div>
<p>In this array, I had to split at the point where the value falls more than 10 counts.</p>
<p>The python snippet which uses numpy to do this</p>
<div class="highlight"><pre><span></span><code><span class="c1"># find the difference between each numbers</span>
<span class="n">diff_values</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">diff</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
<span class="c1"># find the index whose difference is 10</span>
<span class="n">split_index</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">diff_values</span> <span class="o">&</span><span class="n">lt</span><span class="p">;</span> <span class="o">-</span><span class="mi">10</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># add 1 to index values</span>
<span class="n">split_index</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c1"># result</span>
<span class="n">split_values</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">values</span><span class="p">,</span> <span class="n">split_index</span><span class="p">)</span>
</code></pre></div>
<p>The resulting <code>split_values</code> would contain two arrays with values</p>
<div class="highlight"><pre><span></span><code><span class="n">split_values</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="mf">112.0</span><span class="p">,</span> <span class="mf">111.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span> <span class="mf">111.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">112.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span> <span class="mf">113.0</span><span class="p">,</span>
<span class="mf">113.0</span><span class="p">,</span> <span class="mf">114.0</span><span class="p">,</span> <span class="mf">114.0</span><span class="p">,</span> <span class="mf">115.0</span><span class="p">,</span> <span class="mf">117.0</span><span class="p">]</span>
<span class="n">split_values</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="mf">102.0</span><span class="p">,</span> <span class="mf">102.0</span><span class="p">,</span> <span class="mf">102.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span>
<span class="mf">103.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">103.0</span><span class="p">,</span> <span class="mf">104.0</span><span class="p">]</span>
</code></pre></div>
<p>Hope this helps.</p>Import postgres table to HDFS using sqoop2014-08-01T16:00:00+05:302014-08-01T16:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-08-01:/blog/import-postgres-to-hdfs-sqoop.html<p>Importing data from postgres tables into HDFS using <a href="http://sqoop.apache.org">sqoop</a> could be done with the following steps.</p>
<p>Make sure postgres jdbc connector is available in <code>/usr/share/java</code> directory.</p>
<p>To list all available tables in the postgres database</p>
<div class="highlight"><pre><span></span><code>$ sqoop list-tables <span class="se">\</span>
--connect jdbc:postgresql://hostname.com/databaseName <span class="se">\</span>
--username myUserName <span class="se">\</span>
--password myPassword
</code></pre></div>
<p>To …</p><p>Importing data from postgres tables into HDFS using <a href="http://sqoop.apache.org">sqoop</a> could be done with the following steps.</p>
<p>Make sure postgres jdbc connector is available in <code>/usr/share/java</code> directory.</p>
<p>To list all available tables in the postgres database</p>
<div class="highlight"><pre><span></span><code>$ sqoop list-tables <span class="se">\</span>
--connect jdbc:postgresql://hostname.com/databaseName <span class="se">\</span>
--username myUserName <span class="se">\</span>
--password myPassword
</code></pre></div>
<p>To import table to HDFS</p>
<div class="highlight"><pre><span></span><code>$ sqoop import <span class="se">\</span>
--connect jdbc:postgresql://hostname.com/databaseName <span class="se">\</span>
--username myUserName <span class="se">\</span>
--password myPassword <span class="se">\</span>
--table myTableName
--columns <span class="s2">"col_1,col_2"</span>
-m <span class="m">1</span> <span class="se">\</span>
--target-dir /user/hue/myTableName <span class="se">\</span>
--enclosed-by <span class="s1">'\"'</span>
</code></pre></div>
<p>After successfull completion of the above command, the imported file should be available in the target-dir <code>/user/hue/myTableName/part-m-00000</code></p>
<p>This file can be used to create <a href="http://hortonworks.com/hadoop/hcatalog/">HCatalog</a> table for querying with Hive or Pig.</p>
<p>I'm using <a href="http://hortonworks.com">Hortonworks</a> HDPv2.1.1 in AWS EC2.</p>
<p>This is a simple usecase of using sqoop. Hope this helps.</p>Visualization of PyCon 2014 proposals2014-07-10T11:00:00+05:302014-07-10T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-07-10:/blog/visualization-of-pycon-2014-proposals.html<p>Few days back I saw <a href="https://twitter.com/shrayasr">@shrayasr</a> tweet about the number of proposals for <a href="http://in.pycon.org/2014/">PyCon India 2014</a> and his solution to help filter the proposals.</p>
<blockquote class="twitter-tweet" lang="en"><p>I wrote about the issue I faced when looking through <a href="https://twitter.com/hashtag/pyconindia2014?src=hash">#pyconindia2014</a> funnel: <a href="http://t.co/9Qo6DoCjkF">http://t.co/9Qo6DoCjkF</a></p>— Shrayas Rajagopal (@shrayasr) <a href="https://twitter.com/shrayasr/statuses/472783591324139521">May 31, 2014</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" lang="en"><p>Made the <a href="https://twitter.com/hashtag/pyconindia2014?src=hash">#pyconindia2014</a> funnel …</p></blockquote><p>Few days back I saw <a href="https://twitter.com/shrayasr">@shrayasr</a> tweet about the number of proposals for <a href="http://in.pycon.org/2014/">PyCon India 2014</a> and his solution to help filter the proposals.</p>
<blockquote class="twitter-tweet" lang="en"><p>I wrote about the issue I faced when looking through <a href="https://twitter.com/hashtag/pyconindia2014?src=hash">#pyconindia2014</a> funnel: <a href="http://t.co/9Qo6DoCjkF">http://t.co/9Qo6DoCjkF</a></p>— Shrayas Rajagopal (@shrayasr) <a href="https://twitter.com/shrayasr/statuses/472783591324139521">May 31, 2014</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet" lang="en"><p>Made the <a href="https://twitter.com/hashtag/pyconindia2014?src=hash">#pyconindia2014</a> funnel filter bookmarklet into a Chrome extension. Find it here: <a href="https://t.co/1PUAuHkm0I">https://t.co/1PUAuHkm0I</a></p>— Shrayas Rajagopal (@shrayasr) <a href="https://twitter.com/shrayasr/statuses/473028577823186944">June 1, 2014</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>I thought to assist the attendees and the organizers further by creating visualizaiton of the proposals submitted on the <a href="http://in.pycon.org/funnel/2014/">Funnel</a> to get to know better about the variety of topics. The visualization was created with <a href="http://d3js.org/">d3js</a>. </p>
<p data-height="600" data-theme-id="0" data-slug-hash="lrLtg" data-default-tab="result" class='codepen'>See the Pen <a href='http://codepen.io/arundhaj/pen/lrLtg/'>pycon_pie</a> by Arunkumar Dharuman (<a href='http://codepen.io/arundhaj'>@arundhaj</a>) on <a href='http://codepen.io'>CodePen</a>.</p>
<script async src="//codepen.io/assets/embed/ei.js"></script>
<p data-height="750" data-theme-id="0" data-slug-hash="uemha" data-default-tab="result" class='codepen'>See the Pen <a href='http://codepen.io/arundhaj/pen/uemha/'>pycon_sunburst</a> by Arunkumar Dharuman (<a href='http://codepen.io/arundhaj'>@arundhaj</a>) on <a href='http://codepen.io'>CodePen</a>.</p>
<script async src="//codepen.io/assets/embed/ei.js"></script>
<p data-height="500" data-theme-id="0" data-slug-hash="wJLtB" data-default-tab="result" class='codepen'>See the Pen <a href='http://codepen.io/arundhaj/pen/wJLtB/'>pycon_topics_bar</a> by Arunkumar Dharuman (<a href='http://codepen.io/arundhaj'>@arundhaj</a>) on <a href='http://codepen.io'>CodePen</a>.</p>
<script async src="//codepen.io/assets/embed/ei.js"></script>
<p>I've scrapped the data from the funnel on 9th July and is available on my <a href="http://media.arundhaj.com/projects/pycon_visualization/pycon_proposals.csv">site</a>, if anyone interested to create some more awesome visualization are free to use it.</p>
<p>Hope you find it useful.</p>
<p><strong>Update(19/07/2014):</strong> It now fetches live data from the funnel through the JSON endpoint. Thanks Shrayas for letting me know.</p>
<p><strong>Update(21/07/2014):</strong> Added scatter plot to show the confirmation status of the topics vs votes received.</p>
<p data-height="600" data-theme-id="0" data-slug-hash="sbqLn" data-default-tab="result" class='codepen'>See the Pen <a href='http://codepen.io/arundhaj/pen/sbqLn/'>pycon_scatter</a> by Arunkumar Dharuman (<a href='http://codepen.io/arundhaj'>@arundhaj</a>) on <a href='http://codepen.io'>CodePen</a>.</p>
<script async src="//codepen.io/assets/embed/ei.js"></script>Categorizing and summing data in d3js2014-06-04T16:00:00+05:302014-06-04T16:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-06-04:/blog/categorizing-and-summing-data-d3js.html<p>Categorizing an array into different buckets in d3js with d3.nest.</p>
<p>For example, consider a dataset that contains an array of integers ranging values from 1 to 200. If we had to group this array into the following four buckets, as defined as;</p>
<ul>
<li>value >= 150, categorize as excess-heat</li>
<li>value >= 140 …</li></ul><p>Categorizing an array into different buckets in d3js with d3.nest.</p>
<p>For example, consider a dataset that contains an array of integers ranging values from 1 to 200. If we had to group this array into the following four buckets, as defined as;</p>
<ul>
<li>value >= 150, categorize as excess-heat</li>
<li>value >= 140, categorize as heat</li>
<li>value >= 130, categorize as almost-heat</li>
<li>value < 130, categorize as cool</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="c1">// Initializing an array with some dummy values</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">heatArray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">d3</span><span class="p">.</span><span class="nx">shuffle</span><span class="p">(</span><span class="nx">d3</span><span class="p">.</span><span class="nx">range</span><span class="p">(</span><span class="mf">200</span><span class="p">));</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">heatIntermediateArray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">d3</span><span class="p">.</span><span class="nx">nest</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">key</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">d</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mf">150</span><span class="w"> </span><span class="o">?</span><span class="w"></span>
<span class="w"> </span><span class="s2">"excess-heat"</span><span class="w"> </span><span class="o">:</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mf">140</span><span class="w"> </span><span class="o">?</span><span class="w"></span>
<span class="w"> </span><span class="s2">"heat"</span><span class="w"> </span><span class="o">:</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="nx">d</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mf">130</span><span class="w"> </span><span class="o">?</span><span class="w"></span>
<span class="w"> </span><span class="s2">"almost-heat"</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s2">"cool"</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">rollup</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}).</span><span class="nx">entries</span><span class="p">(</span><span class="nx">heatArray</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>The intermediate array generated in the previous step contains the categorized array of four buckets with the count in each bucket.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>
<span class="o">{</span> <span class="s2">"key"</span> : <span class="s2">"excess-heat"</span>, <span class="s2">"values"</span> : <span class="m">50</span><span class="o">}</span>,
<span class="o">{</span> <span class="s2">"key"</span> : <span class="s2">"cool"</span>, <span class="s2">"values"</span> : <span class="m">130</span><span class="o">}</span>,
<span class="o">{</span> <span class="s2">"key"</span> : <span class="s2">"almost-heat"</span>, <span class="s2">"values"</span> : <span class="m">10</span><span class="o">}</span>,
<span class="o">{</span> <span class="s2">"key"</span> : <span class="s2">"heat"</span>, <span class="s2">"values"</span> : <span class="m">10</span><span class="o">}</span>
<span class="o">]</span>
</code></pre></div>
<p>If the required format is something different, it could be remapped with the custom label.</p>
<div class="highlight"><pre><span></span><code><span class="kd">var</span><span class="w"> </span><span class="nx">heatBucket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">heatIntermediateArray</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">{</span><span class="s2">"label"</span><span class="o">:</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="s2">"value"</span><span class="o">:</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">values</span><span class="p">}</span><span class="w"></span>
<span class="p">});</span><span class="w"></span>
</code></pre></div>
<p>And the final output would be of the following format.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span>
<span class="o">{</span> <span class="s2">"label"</span> : <span class="s2">"excess-heat"</span>, <span class="s2">"value"</span> : <span class="m">50</span> <span class="o">}</span>,
<span class="o">{</span> <span class="s2">"label"</span> : <span class="s2">"cool"</span>, <span class="s2">"value"</span> : <span class="m">130</span> <span class="o">}</span>,
<span class="o">{</span> <span class="s2">"label"</span> : <span class="s2">"almost-heat"</span>, <span class="s2">"value"</span> : <span class="m">10</span> <span class="o">}</span>,
<span class="o">{</span> <span class="s2">"label"</span> : <span class="s2">"heat"</span>, <span class="s2">"value"</span> : <span class="m">10</span> <span class="o">}</span>,
<span class="o">]</span>
</code></pre></div>
<p>Hope this helps.</p>Connect remote postgresql server with pgAdmin2014-05-30T15:00:00+05:302014-05-30T15:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-05-30:/blog/connect-remote-postgresql-server-pgadmin.html<p>When trying to connect remote postgresql server with pgAdmin, leads to "Unable to connect" error.</p>
<p>To enable pgAdmin to connect, edit the following configuration in the server:</p>
<div class="highlight"><pre><span></span><code>$ sudo vim /etc/postgresql/9.3/main/postgresql.conf
<span class="nv">listen_address</span> <span class="o">=</span> <span class="s1">'*'</span>
$ sudo vim /etc/postgresql/9.3/main/pg_hba.conf
<span class="nb">local</span> all postgresql trust …</code></pre></div><p>When trying to connect remote postgresql server with pgAdmin, leads to "Unable to connect" error.</p>
<p>To enable pgAdmin to connect, edit the following configuration in the server:</p>
<div class="highlight"><pre><span></span><code>$ sudo vim /etc/postgresql/9.3/main/postgresql.conf
<span class="nv">listen_address</span> <span class="o">=</span> <span class="s1">'*'</span>
$ sudo vim /etc/postgresql/9.3/main/pg_hba.conf
<span class="nb">local</span> all postgresql trust
host all all <span class="m">0</span>.0.0.0/0 md5
$ sudo service postgresql restart
</code></pre></div>
<p>Hope this helps.</p>Single node cluster with Hortonworks HDP2 on EC22014-05-27T16:00:00+05:302014-05-27T16:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-05-27:/blog/installing-hortonworks-hdp2-on-ec2.html<p>The following link provides an extensive guide on setting up a multi-node cluser with <a href="http://hortonworks.com/">Hortonworks</a> HDP2 on <a href="http://aws.amazon.com/">EC2</a>.</p>
<p>Multi-node cluster <a href="http://hortonworks.com/blog/deploying-hadoop-cluster-amazon-ec2-hortonworks">installation guide</a></p>
<p>Though setting up single-node cluster from this guide is quite obvious, I'm documenting here those few so called obvious deviations.</p>
<p>The operating system I choose to install is …</p><p>The following link provides an extensive guide on setting up a multi-node cluser with <a href="http://hortonworks.com/">Hortonworks</a> HDP2 on <a href="http://aws.amazon.com/">EC2</a>.</p>
<p>Multi-node cluster <a href="http://hortonworks.com/blog/deploying-hadoop-cluster-amazon-ec2-hortonworks">installation guide</a></p>
<p>Though setting up single-node cluster from this guide is quite obvious, I'm documenting here those few so called obvious deviations.</p>
<p>The operating system I choose to install is 64-bit RHEL-6.5_GA-x86_64-7-Hourly2 (ami-8d756fe4) and the instance type as m3.large</p>
<p>and proceeding with the guide as is...</p>
<p>For configuring the password-less SSH login for user ec2-user</p>
<div class="highlight"><pre><span></span><code>$ ssh-keygen -t rsa -P <span class="s2">""</span>
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
</code></pre></div>
<p>Once done this, skip the steps to create other nodes. Since we are going to use only single-node.</p>
<p>Proceed to download ambari.repo, install and start it as shown in the guide. Then launch the browser for ambari's cluster install wizard...</p>
<p>For the Target Hosts specify the same node's private DNS address (single line only).</p>
<p>In the same page, for configuring private key choose <code>.pem</code> file, fill user name as ec2-user.</p>
<p>Continue further and finish the steps as mentioned in the guide... And you are done...</p>
<p>Hope this helps.</p>SSLProtocolException handshake alert2014-03-27T16:00:00+05:302014-03-27T16:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-03-27:/blog/sslprotocolexception-handshake-alert.html<p>I was getting the below mentioned exception on my Java client which is trying to establish URL connection to a server.</p>
<div class="highlight"><pre><span></span><code>javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name
</code></pre></div>
<p>The Java client was actually trying to get an XML from the the URL and store it locally in a file as …</p><p>I was getting the below mentioned exception on my Java client which is trying to establish URL connection to a server.</p>
<div class="highlight"><pre><span></span><code>javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name
</code></pre></div>
<p>The Java client was actually trying to get an XML from the the URL and store it locally in a file as;</p>
<div class="highlight"><pre><span></span><code>org.apache.commons.io.FileUtils.copyURLToFile<span class="o">(</span>serverURL, <span class="s2">"localFileName.xml"</span><span class="o">)</span><span class="p">;</span>
</code></pre></div>
<p>It turns out that the server when using multiple hostnames behind a shared IP, it doesn't know what certificate to send. Hence the client expecting a valid certificate fails with SSLProtocolException.</p>
<p>For apache, adding ServerName and ServerAlias to the config would sort out the issue.</p>
<div class="highlight"><pre><span></span><code><VirtualHost mydomain.com:443>
ServerName mydomain.com
ServerAlias www.mydomain.com
</code></pre></div>
<p>If it is a 3rd party server and don't have control, it could even be solved from the client side too. Setting the following system property in your application would do the trick.</p>
<div class="highlight"><pre><span></span><code>System.setProperty<span class="o">(</span><span class="s2">"jsse.enableSNIExtension"</span>, <span class="s2">"false"</span><span class="o">)</span><span class="p">;</span>
</code></pre></div>
<p>Disabling it from the client wouldn't necessarily compromise the security of the application.</p>
<p>More discussion about this issue in <a href="http://stackoverflow.com/questions/7615645/ssl-handshake-alert-unrecognized-name-error-since-upgrade-to-java-1-7-0">StackOverFlow</a></p>PyTypeObject base of Python2014-03-25T15:00:00+05:302014-03-25T15:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-03-25:/blog/pytypeobjects.html<p>PyTypeObject is a structure that defines the basic building blocks of Python, Types.</p>
<p>The definition as found in <code>Include/Object.h</code> file:</p>
<div class="highlight"><pre><span></span><code><span class="k">typedef</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">_typeobject</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject_VAR_HEAD</span><span class="w"></span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">tp_name</span><span class="p">;</span><span class="w"> </span><span class="cm">/* For printing, in format "&lt;module&gt;.&lt;name&gt;" */</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_ssize_t</span><span class="w"> </span><span class="n">tp_basicsize</span><span class="p">,</span><span class="w"> </span><span class="n">tp_itemsize</span><span class="p">;</span><span class="w"> </span><span class="cm">/* For allocation */</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Methods to implement standard operations */</span><span class="w"></span>
<span class="w"> </span><span class="n">destructor …</span></code></pre></div><p>PyTypeObject is a structure that defines the basic building blocks of Python, Types.</p>
<p>The definition as found in <code>Include/Object.h</code> file:</p>
<div class="highlight"><pre><span></span><code><span class="k">typedef</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">_typeobject</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject_VAR_HEAD</span><span class="w"></span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">tp_name</span><span class="p">;</span><span class="w"> </span><span class="cm">/* For printing, in format "&lt;module&gt;.&lt;name&gt;" */</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_ssize_t</span><span class="w"> </span><span class="n">tp_basicsize</span><span class="p">,</span><span class="w"> </span><span class="n">tp_itemsize</span><span class="p">;</span><span class="w"> </span><span class="cm">/* For allocation */</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Methods to implement standard operations */</span><span class="w"></span>
<span class="w"> </span><span class="n">destructor</span><span class="w"> </span><span class="n">tp_dealloc</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">printfunc</span><span class="w"> </span><span class="n">tp_print</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">getattrfunc</span><span class="w"> </span><span class="n">tp_getattr</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">setattrfunc</span><span class="w"> </span><span class="n">tp_setattr</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">cmpfunc</span><span class="w"> </span><span class="n">tp_compare</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">reprfunc</span><span class="w"> </span><span class="n">tp_repr</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Method suites for standard classes */</span><span class="w"></span>
<span class="w"> </span><span class="n">PyNumberMethods</span><span class="w"> </span><span class="o">*</span><span class="n">tp_as_number</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">PySequenceMethods</span><span class="w"> </span><span class="o">*</span><span class="n">tp_as_sequence</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">PyMappingMethods</span><span class="w"> </span><span class="o">*</span><span class="n">tp_as_mapping</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* More standard operations (here for binary compatibility) */</span><span class="w"></span>
<span class="w"> </span><span class="n">hashfunc</span><span class="w"> </span><span class="n">tp_hash</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">ternaryfunc</span><span class="w"> </span><span class="n">tp_call</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">reprfunc</span><span class="w"> </span><span class="n">tp_str</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">getattrofunc</span><span class="w"> </span><span class="n">tp_getattro</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">setattrofunc</span><span class="w"> </span><span class="n">tp_setattro</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Functions to access object as input/output buffer */</span><span class="w"></span>
<span class="w"> </span><span class="n">PyBufferProcs</span><span class="w"> </span><span class="o">*</span><span class="n">tp_as_buffer</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Flags to define presence of optional/expanded features */</span><span class="w"></span>
<span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">tp_flags</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">tp_doc</span><span class="p">;</span><span class="w"> </span><span class="cm">/* Documentation string */</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Assigned meaning in release 2.0 */</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* call function for all accessible objects */</span><span class="w"></span>
<span class="w"> </span><span class="n">traverseproc</span><span class="w"> </span><span class="n">tp_traverse</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* delete references to contained objects */</span><span class="w"></span>
<span class="w"> </span><span class="n">inquiry</span><span class="w"> </span><span class="n">tp_clear</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Assigned meaning in release 2.1 */</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* rich comparisons */</span><span class="w"></span>
<span class="w"> </span><span class="n">richcmpfunc</span><span class="w"> </span><span class="n">tp_richcompare</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* weak reference enabler */</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_ssize_t</span><span class="w"> </span><span class="n">tp_weaklistoffset</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Added in release 2.2 */</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Iterators */</span><span class="w"></span>
<span class="w"> </span><span class="n">getiterfunc</span><span class="w"> </span><span class="n">tp_iter</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">iternextfunc</span><span class="w"> </span><span class="n">tp_iternext</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Attribute descriptor and subclassing stuff */</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">PyMethodDef</span><span class="w"> </span><span class="o">*</span><span class="n">tp_methods</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">PyMemberDef</span><span class="w"> </span><span class="o">*</span><span class="n">tp_members</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">PyGetSetDef</span><span class="w"> </span><span class="o">*</span><span class="n">tp_getset</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">_typeobject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_base</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_dict</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">descrgetfunc</span><span class="w"> </span><span class="n">tp_descr_get</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">descrsetfunc</span><span class="w"> </span><span class="n">tp_descr_set</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_ssize_t</span><span class="w"> </span><span class="n">tp_dictoffset</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">initproc</span><span class="w"> </span><span class="n">tp_init</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">allocfunc</span><span class="w"> </span><span class="n">tp_alloc</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">newfunc</span><span class="w"> </span><span class="n">tp_new</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">freefunc</span><span class="w"> </span><span class="n">tp_free</span><span class="p">;</span><span class="w"> </span><span class="cm">/* Low-level free-memory routine */</span><span class="w"></span>
<span class="w"> </span><span class="n">inquiry</span><span class="w"> </span><span class="n">tp_is_gc</span><span class="p">;</span><span class="w"> </span><span class="cm">/* For PyObject_IS_GC */</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_bases</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_mro</span><span class="p">;</span><span class="w"> </span><span class="cm">/* method resolution order */</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_cache</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_subclasses</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_weaklist</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">destructor</span><span class="w"> </span><span class="n">tp_del</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="cm">/* Type attribute cache version tag. Added in version 2.6 */</span><span class="w"></span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">tp_version_tag</span><span class="p">;</span><span class="w"></span>
<span class="cp">#ifdef COUNT_ALLOCS</span>
<span class="w"> </span><span class="cm">/* these must be last and never explicitly initialized */</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_ssize_t</span><span class="w"> </span><span class="n">tp_allocs</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_ssize_t</span><span class="w"> </span><span class="n">tp_frees</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_ssize_t</span><span class="w"> </span><span class="n">tp_maxalloc</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">_typeobject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_prev</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">_typeobject</span><span class="w"> </span><span class="o">*</span><span class="n">tp_next</span><span class="p">;</span><span class="w"></span>
<span class="cp">#endif</span>
<span class="p">}</span><span class="w"> </span><span class="n">PyTypeObject</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Each types should create an instance of this structure by initializing with appropriate functions pointers as required by the types.</p>
<p>For example, int type is defined as below in <code>Objects/intobject.c</code></p>
<div class="highlight"><pre><span></span><code><span class="n">PyTypeObject</span><span class="w"> </span><span class="n">PyInt_Type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">PyVarObject_HEAD_INIT</span><span class="p">(</span><span class="o">&</span><span class="n">PyType_Type</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="s">"int"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">PyIntObject</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">destructor</span><span class="p">)</span><span class="n">int_dealloc</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_dealloc */</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">printfunc</span><span class="p">)</span><span class="n">int_print</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_print */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_getattr */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_setattr */</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">cmpfunc</span><span class="p">)</span><span class="n">int_compare</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_compare */</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">reprfunc</span><span class="p">)</span><span class="n">int_to_decimal_string</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_repr */</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="n">int_as_number</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_as_number */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_as_sequence */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_as_mapping */</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">hashfunc</span><span class="p">)</span><span class="n">int_hash</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_hash */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_call */</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">reprfunc</span><span class="p">)</span><span class="n">int_to_decimal_string</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_str */</span><span class="w"></span>
<span class="w"> </span><span class="n">PyObject_GenericGetAttr</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_getattro */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_setattro */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_as_buffer */</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_TPFLAGS_DEFAULT</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Py_TPFLAGS_CHECKTYPES</span><span class="w"> </span><span class="o">|</span><span class="w"></span>
<span class="w"> </span><span class="n">Py_TPFLAGS_BASETYPE</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Py_TPFLAGS_INT_SUBCLASS</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_flags */</span><span class="w"></span>
<span class="w"> </span><span class="n">int_doc</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_doc */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_traverse */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_clear */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_richcompare */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_weaklistoffset */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_iter */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_iternext */</span><span class="w"></span>
<span class="w"> </span><span class="n">int_methods</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_methods */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_members */</span><span class="w"></span>
<span class="w"> </span><span class="n">int_getset</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_getset */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_base */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_dict */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_descr_get */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_descr_set */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_dictoffset */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_init */</span><span class="w"></span>
<span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_alloc */</span><span class="w"></span>
<span class="w"> </span><span class="n">int_new</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_new */</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">freefunc</span><span class="p">)</span><span class="n">int_free</span><span class="p">,</span><span class="w"> </span><span class="cm">/* tp_free */</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</code></pre></div>Starting Core Python Programming Book2014-03-18T10:00:00+05:302014-03-18T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2014-03-18:/blog/starting-core-python-programming.html<p>I'm starting to read the <a href="http://corepython.com">Core Python Programming</a> book. Over past few months I've already started to program in python, like developing web apps with <a href="https://www.djangoproject.com">django</a> and data analytics with <a href="http://pandas.pydata.org">pandas</a>. I started to like the language, so wanted to go back to the basics of it and I'm begining …</p><p>I'm starting to read the <a href="http://corepython.com">Core Python Programming</a> book. Over past few months I've already started to program in python, like developing web apps with <a href="https://www.djangoproject.com">django</a> and data analytics with <a href="http://pandas.pydata.org">pandas</a>. I started to like the language, so wanted to go back to the basics of it and I'm begining with this book. I've already used C, C++, Java, C# and JavaScript, now I'm excited to add python to the list.</p>
<p>I wanted to attempt to blog as I read the book, and obviously the first blog is reserved for "The Zen of Python" by Tim Peters as stated in <a href="http://legacy.python.org/dev/peps/pep-0020">PEP 20</a>.</p>
<div class="highlight"><pre><span></span><code><span class="n">Beautiful</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="n">ugly</span><span class="p">.</span><span class="w"></span>
<span class="n">Explicit</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="n">implicit</span><span class="p">.</span><span class="w"></span>
<span class="n">Simple</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="n">complex</span><span class="p">.</span><span class="w"></span>
<span class="n">Complex</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="n">complicated</span><span class="p">.</span><span class="w"></span>
<span class="n">Flat</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="n">nested</span><span class="p">.</span><span class="w"></span>
<span class="n">Sparse</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="n">dense</span><span class="p">.</span><span class="w"></span>
<span class="n">Readability</span><span class="w"> </span><span class="n">counts</span><span class="p">.</span><span class="w"></span>
<span class="n">Special</span><span class="w"> </span><span class="n">cases</span><span class="w"> </span><span class="n">aren</span><span class="c">'t special enough to break the rules.</span><span class="w"></span>
<span class="n">Although</span><span class="w"> </span><span class="n">practicality</span><span class="w"> </span><span class="n">beats</span><span class="w"> </span><span class="n">purity</span><span class="p">.</span><span class="w"></span>
<span class="n">Errors</span><span class="w"> </span><span class="n">should</span><span class="w"> </span><span class="n">never</span><span class="w"> </span><span class="n">pass</span><span class="w"> </span><span class="n">silently</span><span class="p">.</span><span class="w"></span>
<span class="n">Unless</span><span class="w"> </span><span class="n">explicitly</span><span class="w"> </span><span class="n">silenced</span><span class="p">.</span><span class="w"></span>
<span class="ow">In</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">face</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">ambiguity</span><span class="p">,</span><span class="w"> </span><span class="n">refuse</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">temptation</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">guess</span><span class="p">.</span><span class="w"></span>
<span class="n">There</span><span class="w"> </span><span class="n">should</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">one</span><span class="o">--</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">preferably</span><span class="w"> </span><span class="n">only</span><span class="w"> </span><span class="n">one</span><span class="w"> </span><span class="o">--</span><span class="n">obvious</span><span class="w"> </span><span class="n">way</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="n">it</span><span class="p">.</span><span class="w"></span>
<span class="n">Although</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">way</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">obvious</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">first</span><span class="w"> </span><span class="n">unless</span><span class="w"> </span><span class="n">you</span><span class="c">'re Dutch.</span><span class="w"></span>
<span class="n">Now</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="n">never</span><span class="p">.</span><span class="w"></span>
<span class="n">Although</span><span class="w"> </span><span class="n">never</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">often</span><span class="w"> </span><span class="n">better</span><span class="w"> </span><span class="n">than</span><span class="w"> </span><span class="o">*</span><span class="n">right</span><span class="o">*</span><span class="w"> </span><span class="n">now</span><span class="p">.</span><span class="w"></span>
<span class="k">If</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">implementation</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">hard</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">explain</span><span class="p">,</span><span class="w"> </span><span class="n">it</span><span class="c">'s a bad idea.</span><span class="w"></span>
<span class="k">If</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">implementation</span><span class="w"> </span><span class="ow">is</span><span class="w"> </span><span class="n">easy</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">explain</span><span class="p">,</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">good</span><span class="w"> </span><span class="n">idea</span><span class="p">.</span><span class="w"></span>
<span class="n">Namespaces</span><span class="w"> </span><span class="n">are</span><span class="w"> </span><span class="n">one</span><span class="w"> </span><span class="n">honking</span><span class="w"> </span><span class="n">great</span><span class="w"> </span><span class="n">idea</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="k">let</span><span class="c">'s do more of those!</span><span class="w"></span>
</code></pre></div>Pandas with Postgres datasource2013-12-19T12:00:00+05:302013-12-19T12:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-12-19:/blog/pandas-with-postgres-datasource.html<p>We would be getting data from different sources for doing data analysis. Most common being from files, we might even source from databases.</p>
<p>This post will show how to load data from <a href="http://www.postgresql.org">postgres</a> database to <a href="http://pandas.pydata.org">pandas</a> DataFrame. pandas is a python based data analysis tool.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">psycopg2</span> <span class="k">as</span> <span class="nn">pg</span>
<span class="kn">import …</span></code></pre></div><p>We would be getting data from different sources for doing data analysis. Most common being from files, we might even source from databases.</p>
<p>This post will show how to load data from <a href="http://www.postgresql.org">postgres</a> database to <a href="http://pandas.pydata.org">pandas</a> DataFrame. pandas is a python based data analysis tool.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">psycopg2</span> <span class="k">as</span> <span class="nn">pg</span>
<span class="kn">import</span> <span class="nn">pandas.io.sql</span> <span class="k">as</span> <span class="nn">psql</span>
<span class="c1"># get connected to the database</span>
<span class="n">connection</span> <span class="o">=</span> <span class="n">pg</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">"dbname=mydatabase user=postgres"</span><span class="p">)</span>
<span class="n">dataframe</span> <span class="o">=</span> <span class="n">psql</span><span class="o">.</span><span class="n">frame_query</span><span class="p">(</span><span class="s2">"SELECT id, price FROM stock_price"</span><span class="p">,</span> <span class="n">connection</span><span class="p">)</span>
</code></pre></div>
<p>Make sure the packages are installed. </p>
<p>This simple snippet would come handy at times. </p>Spring Hibernate - Create table if not exists2013-10-25T09:00:00+05:302013-10-25T09:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-10-25:/blog/spring-hibernate-create-table-if-not-exists.html<p>We have to configure hibernate, if we require the application to create the table when it doesn't already exists. Else, we would end up with the below mentioned exception.</p>
<div class="highlight"><pre><span></span><code>ERROR: ERROR: relation <span class="s2">"mytable"</span> does not exist
Position: <span class="m">13</span>
org.hibernate.exception.SQLGrammarException: could not execute statement
at org.hibernate.exception …</code></pre></div><p>We have to configure hibernate, if we require the application to create the table when it doesn't already exists. Else, we would end up with the below mentioned exception.</p>
<div class="highlight"><pre><span></span><code>ERROR: ERROR: relation <span class="s2">"mytable"</span> does not exist
Position: <span class="m">13</span>
org.hibernate.exception.SQLGrammarException: could not execute statement
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert<span class="o">(</span>SQLStateConversionDelegate.java:122<span class="o">)</span>
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert<span class="o">(</span>StandardSQLExceptionConverter.java:49<span class="o">)</span>
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert<span class="o">(</span>SqlExceptionHelper.java:125<span class="o">)</span>
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert<span class="o">(</span>SqlExceptionHelper.java:110<span class="o">)</span>
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate<span class="o">(</span>ResultSetReturnImpl.java:136<span class="o">)</span>
at org.hibernate.id.IdentityGenerator<span class="nv">$GetGeneratedKeysDelegate</span>.executeAndExtract<span class="o">(</span>IdentityGenerator.java:96<span class="o">)</span>
at org.hibernate.id.insert.AbstractReturningDelegate.performInsert<span class="o">(</span>AbstractReturningDelegate.java:58<span class="o">)</span>
at org.hibernate.persister.entity.AbstractEntityPersister.insert<span class="o">(</span>AbstractEntityPersister.java:2966<span class="o">)</span>
at org.hibernate.persister.entity.AbstractEntityPersister.insert<span class="o">(</span>AbstractEntityPersister.java:3477<span class="o">)</span>
at org.hibernate.action.internal.EntityIdentityInsertAction.execute<span class="o">(</span>EntityIdentityInsertAction.java:81<span class="o">)</span>
at org.hibernate.engine.spi.ActionQueue.execute<span class="o">(</span>ActionQueue.java:362<span class="o">)</span>
at org.hibernate.engine.spi.ActionQueue.addResolvedEntityInsertAction<span class="o">(</span>ActionQueue.java:203<span class="o">)</span>
at org.hibernate.engine.spi.ActionQueue.addInsertAction<span class="o">(</span>ActionQueue.java:183<span class="o">)</span>
at org.hibernate.engine.spi.ActionQueue.addAction<span class="o">(</span>ActionQueue.java:167<span class="o">)</span>
at org.hibernate.event.internal.AbstractSaveEventListener.addInsertAction<span class="o">(</span>AbstractSaveEventListener.java:321<span class="o">)</span>
at org.hibernate.event.internal.AbstractSaveEventListener.performSaveOrReplicate<span class="o">(</span>AbstractSaveEventListener.java:286<span class="o">)</span>
at org.hibernate.event.internal.AbstractSaveEventListener.performSave<span class="o">(</span>AbstractSaveEventListener.java:192<span class="o">)</span>
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId<span class="o">(</span>AbstractSaveEventListener.java:125<span class="o">)</span>
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId<span class="o">(</span>DefaultSaveOrUpdateEventListener.java:204<span class="o">)</span>
at org.hibernate.event.internal.DefaultSaveEventListener.saveWithGeneratedOrRequestedId<span class="o">(</span>DefaultSaveEventListener.java:55<span class="o">)</span>
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsTransient<span class="o">(</span>DefaultSaveOrUpdateEventListener.java:189<span class="o">)</span>
at org.hibernate.event.internal.DefaultSaveEventListener.performSaveOrUpdate<span class="o">(</span>DefaultSaveEventListener.java:49<span class="o">)</span>
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate<span class="o">(</span>DefaultSaveOrUpdateEventListener.java:90<span class="o">)</span>
at org.hibernate.internal.SessionImpl.fireSave<span class="o">(</span>SessionImpl.java:763<span class="o">)</span>
at org.hibernate.internal.SessionImpl.save<span class="o">(</span>SessionImpl.java:755<span class="o">)</span>
at org.hibernate.internal.SessionImpl.save<span class="o">(</span>SessionImpl.java:751<span class="o">)</span>
at com.arundhaj.domain.MyTableDao.save<span class="o">(</span>MyTableDao.java:17<span class="o">)</span>
at sun.reflect.NativeMethodAccessorImpl.invoke0<span class="o">(</span>Native Method<span class="o">)</span>
at sun.reflect.NativeMethodAccessorImpl.invoke<span class="o">(</span>NativeMethodAccessorImpl.java:57<span class="o">)</span>
at sun.reflect.DelegatingMethodAccessorImpl.invoke<span class="o">(</span>DelegatingMethodAccessorImpl.java:43<span class="o">)</span>
at java.lang.reflect.Method.invoke<span class="o">(</span>Method.java:606<span class="o">)</span>
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection<span class="o">(</span>AopUtils.java:317<span class="o">)</span>
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint<span class="o">(</span>ReflectiveMethodInvocation.java:183<span class="o">)</span>
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed<span class="o">(</span>ReflectiveMethodInvocation.java:150<span class="o">)</span>
at org.springframework.transaction.interceptor.TransactionInterceptor<span class="nv">$1</span>.proceedWithInvocation<span class="o">(</span>TransactionInterceptor.java:96<span class="o">)</span>
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction<span class="o">(</span>TransactionAspectSupport.java:260<span class="o">)</span>
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke<span class="o">(</span>TransactionInterceptor.java:94<span class="o">)</span>
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed<span class="o">(</span>ReflectiveMethodInvocation.java:172<span class="o">)</span>
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke<span class="o">(</span>JdkDynamicAopProxy.java:204<span class="o">)</span>
at com.sun.proxy.<span class="nv">$Proxy15</span>.save<span class="o">(</span>Unknown Source<span class="o">)</span>
at com.arundhaj.domain.MyTableBo.save<span class="o">(</span>MyTableBo.java:15<span class="o">)</span>
at com.arundhaj.Evaluate.run<span class="o">(</span>Evaluate.java:38<span class="o">)</span>
at com.arundhaj.App.main<span class="o">(</span>App.java:22<span class="o">)</span>
Caused by: org.postgresql.util.PSQLException: ERROR: relation <span class="s2">"mytable"</span> does not exist
Position: <span class="m">13</span>
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse<span class="o">(</span>QueryExecutorImpl.java:2103<span class="o">)</span>
at org.postgresql.core.v3.QueryExecutorImpl.processResults<span class="o">(</span>QueryExecutorImpl.java:1836<span class="o">)</span>
at org.postgresql.core.v3.QueryExecutorImpl.execute<span class="o">(</span>QueryExecutorImpl.java:257<span class="o">)</span>
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute<span class="o">(</span>AbstractJdbc2Statement.java:512<span class="o">)</span>
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags<span class="o">(</span>AbstractJdbc2Statement.java:388<span class="o">)</span>
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate<span class="o">(</span>AbstractJdbc2Statement.java:334<span class="o">)</span>
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate<span class="o">(</span>ResultSetReturnImpl.java:133<span class="o">)</span>
... <span class="m">38</span> more
</code></pre></div>
<p>The required hibernate property in the bean configuration file is.</p>
<div class="highlight"><pre><span></span><code> <span class="nt"><property</span> <span class="na">name=</span><span class="s">"hibernateProperties"</span><span class="nt">></span>
<span class="nt"><props></span>
<span class="nt"><prop</span> <span class="na">key=</span><span class="s">"hibernate.dialect"</span><span class="nt">></span>org.hibernate.dialect.PostgreSQLDialect<span class="nt"></prop></span>
<span class="nt"><prop</span> <span class="na">key=</span><span class="s">"hibernate.show_sql"</span><span class="nt">></span>true<span class="nt"></prop></span>
<span class="nt"><prop</span> <span class="na">key=</span><span class="s">"hibernate.hbm2ddl"</span><span class="nt">></span>create<span class="nt"></prop></span>
<span class="nt"></props></span>
<span class="nt"></property></span>
</code></pre></div>
<p>The configuration <code>hibernate.hbm2ddl</code> would <code>create</code> a table, if it doesn't already exits. However, every time when we run the application, it will always create a new table by deleting the existing table. So, the value should be changed to <code>update</code>, if you wanted to create the table only once and from next time, use the available table.</p>
<p>Hope it would help the starters getting hands on hibernate.</p>IMDB top 250 visualized with d3js2013-09-07T09:00:00+05:302013-09-07T09:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-09-07:/blog/imdb-top250-visualized-with-d3js.html<p>IMDB top 250 movies visualized with d3js</p><script src="http://media.arundhaj.com/blog/imdb-top250/imdb1.js" type="text/javascript" ></script>
<p>IMDB <a href="http://www.imdb.com/chart/top">Top 250</a> movies visualized with <a href="http://d3js.org">d3js</a>.</p>
<p>This was inspired by a <a href="http://in.pycon.org/funnel/2013/73-data-visualisation-in-powerpoint-with-python">Data visualization Session</a> in <a href="http://in.pycon.org/2013">PyCon India 2013</a> presented by <a href="http://www.s-anand.net/">Anand S</a></p>
<p>To sort the list by ratings and alphanumeric, click the "Toggle Sorting" button. Source code is uploaded in <a href="https://github.com/arundhaj/d3js_sandbox/tree/master/imdbtop250">github</a></p>
<p><button id="sortToggle">Toggle Sorting</button></p>
<div id="imdb" style="margin-left: 5px; height: 620px; width: 980px;">
<svg />
</div>Starting CDH quickstart VM in Windows2013-09-07T09:00:00+05:302013-09-07T09:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-09-07:/blog/starting-cdh-vm-in-windows.html<p>While trying <a href="http://www.cloudera.com/content/cloudera/en/products/cdh.html">Cloudera CDH</a> quickstart VM in my Windows 7 machine, I was facing issues to start it with <a href="http://www.vmware.com/products/player">VMware player</a>.</p>
<p>I was getting the following error message.</p>
<div class="highlight"><pre><span></span><code>The import failed because D:<span class="se">\V</span>irtualMachines<span class="se">\c</span>loudera-quickstart-vm-4.3.0-vmware<span class="se">\c</span>loudera-quickstart-vm-4.3.0-vmware.ovf did not pass OVF specification conformance …</code></pre></div><p>While trying <a href="http://www.cloudera.com/content/cloudera/en/products/cdh.html">Cloudera CDH</a> quickstart VM in my Windows 7 machine, I was facing issues to start it with <a href="http://www.vmware.com/products/player">VMware player</a>.</p>
<p>I was getting the following error message.</p>
<div class="highlight"><pre><span></span><code>The import failed because D:<span class="se">\V</span>irtualMachines<span class="se">\c</span>loudera-quickstart-vm-4.3.0-vmware<span class="se">\c</span>loudera-quickstart-vm-4.3.0-vmware.ovf did not pass OVF specification conformance or virtual hardware compliance checks.
Click Retry to relax OVF specification and virtual hardware compliance checks and try the import again, or click Cancel to cancel the import. If you retry the import, you might not be able to use the virtual machine <span class="k">in</span> VMware Player.
</code></pre></div>
<p>On selecting to retry, I was further getting the following error message.</p>
<div class="highlight"><pre><span></span><code>Failed to open virtual machine: SHA1 digest of file cloudera-quickstart-vm-4.3.0-vmware.ovf does not match manifest.
</code></pre></div>
<p>Initially, it seemed like a corrupt download. Later with little bit of googling, I found out the solution.</p>
<p>WinZip I was using to extract <code>tar.gz</code> file was not able to extract it. Then by extracting with <a href="http://www.7-zip.org/">7-Zip</a>, I was able to run the VM successfully.</p>Single File Minimal Webapp in Django2013-09-05T11:00:00+05:302013-09-05T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-09-05:/blog/single-file-minimal-webapp-in-django.html<p>Last week I've attended a <a href="http://in.pycon.org/funnel/2013/59-django-beyond-basics">Django Session</a> in <a href="http://in.pycon.org/2013">PyCon India 2013</a> presented by <a href="http://arunrocks.com">Arun Ravindran</a>. It was a good session which covered the basics. I started learning <a href="https://www.djangoproject.com">Django</a> by doing a considerably big application, and now I'm uncovering the basics step by step. In certain cases I required a very …</p><p>Last week I've attended a <a href="http://in.pycon.org/funnel/2013/59-django-beyond-basics">Django Session</a> in <a href="http://in.pycon.org/2013">PyCon India 2013</a> presented by <a href="http://arunrocks.com">Arun Ravindran</a>. It was a good session which covered the basics. I started learning <a href="https://www.djangoproject.com">Django</a> by doing a considerably big application, and now I'm uncovering the basics step by step. In certain cases I required a very small application but ended up creating a whole bunch of files with <code>django-admin.py startproject</code>. For this reason I thought of exploring web.py, flask etc., but was not convinced to move out of Django. This session introduced me a single file django application, which exactly I was looking at. Then I tried it at home.</p>
<p>With a minimal changes to the code shared by Arun, I was successfully able to verify it. And here is the code snippet.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">django.conf.urls.defaults</span> <span class="kn">import</span> <span class="n">patterns</span>
<span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">HttpResponse</span>
<span class="n">filepath</span><span class="p">,</span> <span class="n">extension</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="vm">__file__</span><span class="p">)</span>
<span class="n">DEBUG</span><span class="o">=</span><span class="kc">True</span>
<span class="n">ROOT_URLCONF</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">filepath</span><span class="p">)</span> <span class="c1"># same should be given in settings "index"</span>
<span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'default'</span> <span class="p">:</span> <span class="p">{}}</span>
<span class="n">SECRET_KEY</span> <span class="o">=</span> <span class="s1">'0123456789'</span> <span class="o">*</span> <span class="mi">50</span>
<span class="k">def</span> <span class="nf">index</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="k">return</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="s1">'Hello World!'</span><span class="p">)</span>
<span class="n">urlpatterns</span> <span class="o">=</span> <span class="n">patterns</span><span class="p">(</span><span class="s1">''</span><span class="p">,</span> <span class="p">(</span><span class="sa">r</span><span class="s1">'^$'</span><span class="p">,</span> <span class="n">index</span><span class="p">))</span>
</code></pre></div>
<p>Save the above snippet in a file named, say <code>index.py</code>. Then run the below command to get it started in Windows.</p>
<div class="highlight"><pre><span></span><code>D:<span class="se">\d</span>jango> python D:<span class="se">\P</span>rogramFiles<span class="se">\P</span>ython27<span class="se">\L</span>ib<span class="se">\s</span>ite-packages<span class="se">\d</span>jango<span class="se">\b</span>in<span class="se">\d</span>jango-admin.py runserver <span class="m">8000</span> --pythonpath<span class="o">=</span>. --settings<span class="o">=</span>index
</code></pre></div>
<p>And for Linux</p>
<div class="highlight"><pre><span></span><code>$ <span class="nv">PYTHONPATH</span><span class="o">=</span>. django-admin.py runserver <span class="m">0</span>.0.0.0:8000 --settings<span class="o">=</span>index
</code></pre></div>
<p>However, this doesn't work when we switch off the DEBUG mode. I think that there should be some configurations change that has to be done when running in Apache. I'll update as I find it.</p>Getting Started with ZeroMQ in Python2013-07-24T17:00:00+05:302013-07-24T17:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-07-24:/blog/getting-started-with-zeromq-in-python.html<p>A simple getting started sample for ZeroMQ in Python. We will implement a Echo Server and Echo Client.</p>
<p><code>echo_server.py</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">zmq</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span>
<span class="n">socket</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REP</span><span class="p">)</span>
<span class="n">socket</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="s2">"tcp://127.0.0.1:5000"</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"Got"</span><span class="p">,</span> <span class="n">msg</span>
<span class="n">socket</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
</code></pre></div>
<p><code>echo_client …</code></p><p>A simple getting started sample for ZeroMQ in Python. We will implement a Echo Server and Echo Client.</p>
<p><code>echo_server.py</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">zmq</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span>
<span class="n">socket</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REP</span><span class="p">)</span>
<span class="n">socket</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="s2">"tcp://127.0.0.1:5000"</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"Got"</span><span class="p">,</span> <span class="n">msg</span>
<span class="n">socket</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
</code></pre></div>
<p><code>echo_client.py</code></p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">zmq</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span>
<span class="n">socket</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">REQ</span><span class="p">)</span>
<span class="n">socket</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s2">"tcp://127.0.0.1:5000"</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">raw_input</span><span class="p">(</span><span class="s2">"Enter msg:"</span><span class="p">)</span>
<span class="n">socket</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="n">msg_in</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"GOT"</span><span class="p">,</span> <span class="n">msg_in</span>
</code></pre></div>Converting repeating decimals into fractions2013-07-23T16:00:00+05:302013-07-23T16:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-07-23:/blog/converting-repeating-decimals-into-fractions.html<p>I came across an interesting maths tip, which I couldn't resist sharing.</p>
<p>Turning a fraction to a decimal is a childs' play, with a simple division. The trick is on restoring the fraction back from the decimal that has a repeating numbers.</p>
<p>Let us take a decimal "0.4545454545..."</p>
<p>Pick …</p><p>I came across an interesting maths tip, which I couldn't resist sharing.</p>
<p>Turning a fraction to a decimal is a childs' play, with a simple division. The trick is on restoring the fraction back from the decimal that has a repeating numbers.</p>
<p>Let us take a decimal "0.4545454545..."</p>
<p>Pick the number that repeats, in this case it is "45", and divide it by however many 9's as there are digits in the number. As size of 45 is 2, then take 2 9's, i.e. 99.</p>
<p>Now simplify 45/99, that would lead to 5/11.</p>
<p>and following is the reason why this works...</p>
<p>Consider, K = 0.4545454545...</p>
<p>100K = 100 x 0.4545454545 = 45.45454545...</p>
<p>100K - K = 45.45454545... - 0.4545454545...</p>
<p>99K = 45</p>
<p>K = 45/99 => 5/11</p>
<p>Thats' it.</p>Better not to ask feedback2013-07-11T10:00:00+05:302013-07-11T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-07-11:/blog/better-not-to-ask-feedback.html<div class="highlight"><pre><span></span><code><span class="nv">Better</span><span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">ask</span><span class="w"> </span><span class="nv">feedback</span>,<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nv">you</span><span class="w"> </span><span class="nv">don</span><span class="err">'t respect it.</span><span class="w"></span>
</code></pre></div>
<p>It is the feedback which managers' ask the team. Feedback about work, people, environment and organization. Getting feedback from people generally motivates, as it makes people feel better that they are being heard. </p>
<p>Though it is mandatory for the …</p><div class="highlight"><pre><span></span><code><span class="nv">Better</span><span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">ask</span><span class="w"> </span><span class="nv">feedback</span>,<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nv">you</span><span class="w"> </span><span class="nv">don</span><span class="err">'t respect it.</span><span class="w"></span>
</code></pre></div>
<p>It is the feedback which managers' ask the team. Feedback about work, people, environment and organization. Getting feedback from people generally motivates, as it makes people feel better that they are being heard. </p>
<p>Though it is mandatory for the managers to get feedback, at times it would frustrates people more than they are already frustrated. At the time when they find their feedback was not respected. Respect come from addressing the feedback, amending the value it brings, or atleast reasoning why the feedback was not considered. But ignoring the feedback as if it was never heard, disrespect the feedback and also the person.</p>
<p>So, if the intention to ask feedback is to just motivate the people and not really addressing it; it is better not to ask feedback. </p>Difference between the request time and the current time is too large2013-06-26T14:00:00+05:302013-06-26T14:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-06-26:/blog/difference-between-time-is-too-large.html<p>With <code>s3cmd</code> to get files from S3 to EC2 I was getting this error "The difference between the request time and the current time is too large" and the command got terminated.</p>
<p>Found out that the time on EC2 instance was laging too much and wondered about the cause of …</p><p>With <code>s3cmd</code> to get files from S3 to EC2 I was getting this error "The difference between the request time and the current time is too large" and the command got terminated.</p>
<p>Found out that the time on EC2 instance was laging too much and wondered about the cause of the difference. Searching on web I could make out that the EC2 instance uses a software clock and it introuduces few seconds lag each day, which piles to this huge difference. Then I had to re-set the time to make the file tranfer work.</p>
<p>on command line set the data as below:</p>
<div class="highlight"><pre><span></span><code>sudo date <span class="m">062610252013</span>.20
</code></pre></div>
<p>sets date and time to Jun 26 10:25:20 2013 (MMDDhhmmYYYY.ss)</p>
<p>It can also be done by configuring an NTP Server to auto sync the time. </p>
<p>Hope this helps.</p>Using mail during django development2013-05-09T11:00:00+05:302013-05-09T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-05-09:/blog/using-mail-during-django-development.html<p>While developing a module that requires to send emails, using actual email servers would be an overhead. They might introduce network latency, possible spam filtering etc. All end up retarding productivity.</p>
<p>We have <code>smtpd.DebuggingServer</code> that could be used as a local email server. All emails that are sent to …</p><p>While developing a module that requires to send emails, using actual email servers would be an overhead. They might introduce network latency, possible spam filtering etc. All end up retarding productivity.</p>
<p>We have <code>smtpd.DebuggingServer</code> that could be used as a local email server. All emails that are sent to this server outputs to the stdout. To start the server.</p>
<div class="highlight"><pre><span></span><code>$ python -m smtpd -n -c DebuggingServer localhost:1025
</code></pre></div>
<p>and the corresponding setting to be used in settings.py is</p>
<div class="highlight"><pre><span></span><code><span class="nv">EMAIL_HOST</span> <span class="o">=</span> <span class="s1">'localhost'</span>
<span class="nv">EMAIL_PORT</span> <span class="o">=</span> <span class="m">1025</span>
</code></pre></div>
<p>The code snippet that sends the mail is below.</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">django.core.mail</span> <span class="kn">import</span> <span class="n">EmailMultiAlternatives</span>
<span class="kn">from</span> <span class="nn">django.template.loader</span> <span class="kn">import</span> <span class="n">render_to_string</span>
<span class="kn">from</span> <span class="nn">django.utils.html</span> <span class="kn">import</span> <span class="n">strip_tags</span>
<span class="n">subject</span><span class="p">,</span> <span class="n">from_email</span> <span class="o">=</span> <span class="s1">'Account created'</span><span class="p">,</span> <span class="s1">'admin@mywebsite.com'</span>
<span class="n">html_content</span> <span class="o">=</span> <span class="n">render_to_string</span><span class="p">(</span><span class="s1">'email.html'</span><span class="p">)</span>
<span class="n">text_content</span> <span class="o">=</span> <span class="n">strip_tags</span><span class="p">(</span><span class="n">html_content</span><span class="p">)</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">EmailMultiAlternatives</span><span class="p">(</span><span class="n">subject</span><span class="p">,</span> <span class="n">text_content</span><span class="p">,</span> <span class="n">from_email</span><span class="p">,</span> <span class="p">[</span><span class="n">user</span><span class="nd">@somewebsite</span><span class="o">.</span><span class="n">com</span><span class="p">])</span>
<span class="n">msg</span><span class="o">.</span><span class="n">attach_alternative</span><span class="p">(</span><span class="n">html_content</span><span class="p">,</span> <span class="s1">'text/html'</span><span class="p">)</span>
<span class="n">msg</span><span class="o">.</span><span class="n">send</span><span class="p">()</span>
</code></pre></div>
<p>Upon calling send message, the mail would be delivered to the DebuggingServer and displayed in stdout as below.</p>
<div class="highlight"><pre><span></span><code>---------- MESSAGE FOLLOWS ----------
Content-Type: multipart/alternative<span class="p">;</span> <span class="nv">boundary</span><span class="o">=</span><span class="s2">"===============0236101087=="</span>
MIME-Version: <span class="m">1</span>.0
Subject: Account created
From: admin@mywebsite.com
To: user@somewebsite.com
Date: Thu, <span class="m">09</span> May <span class="m">2013</span> <span class="m">05</span>:53:21 -0000
X-Peer: <span class="m">127</span>.0.0.1
--<span class="o">===============</span><span class="nv">0236101087</span><span class="o">==</span>
Content-Type: text/plain<span class="p">;</span> <span class="nv">charset</span><span class="o">=</span><span class="s2">"utf-8"</span>
MIME-Version: <span class="m">1</span>.0
Content-Transfer-Encoding: 7bit
Mail sent from mywebsite.com
Account created.
--<span class="o">===============</span><span class="nv">0236101087</span><span class="o">==</span>
Content-Type: text/html<span class="p">;</span> <span class="nv">charset</span><span class="o">=</span><span class="s2">"utf-8"</span>
MIME-Version: <span class="m">1</span>.0
Content-Transfer-Encoding: 7bit
<p>Mail sent from mywebsite.com</p>
<div <span class="nv">style</span><span class="o">=</span><span class="s2">"background: #ddd;border: 1px solid #d3d3d3;-moz-border-radius: 3px;border-radius: 3px;width: 50%;line-height: 100px;text-align: center;"</span>>
Account created.
</div>
--<span class="o">===============</span><span class="nv">0236101087</span><span class="o">==</span>--
------------ END MESSAGE ------------
</code></pre></div>
<p>Hope this helps.</p>QuerySyntaxException: Foo is not mapped2013-04-18T12:30:00+05:302013-04-18T12:30:00+05:30ArunDhaJtag:www.arundhaj.com,2013-04-18:/blog/querysyntaxexception-not-mapped.html<p>While writing query using hibernate, I always tend to query based on table name and end up getting <code>QuerySyntaxException</code></p>
<div class="highlight"><pre><span></span><code><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">QuerySyntaxException</span><span class="p">:</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">not</span><span class="w"> </span><span class="n">mapped</span><span class="w"> </span><span class="o">[</span><span class="n">from</span><span class="w"> </span><span class="n">foo</span><span class="o">]</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">SessionFactoryHelper</span><span class="p">.</span><span class="na">requireClassPersister</span><span class="p">(</span><span class="n">SessionFactoryHelper</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">180</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">tree …</span></code></pre></div><p>While writing query using hibernate, I always tend to query based on table name and end up getting <code>QuerySyntaxException</code></p>
<div class="highlight"><pre><span></span><code><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">QuerySyntaxException</span><span class="p">:</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">not</span><span class="w"> </span><span class="n">mapped</span><span class="w"> </span><span class="o">[</span><span class="n">from</span><span class="w"> </span><span class="n">foo</span><span class="o">]</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">SessionFactoryHelper</span><span class="p">.</span><span class="na">requireClassPersister</span><span class="p">(</span><span class="n">SessionFactoryHelper</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">180</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">tree</span><span class="p">.</span><span class="na">FromElementFactory</span><span class="p">.</span><span class="na">addFromElement</span><span class="p">(</span><span class="n">FromElementFactory</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">110</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">tree</span><span class="p">.</span><span class="na">FromClause</span><span class="p">.</span><span class="na">addFromElement</span><span class="p">(</span><span class="n">FromClause</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">93</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">HqlSqlWalker</span><span class="p">.</span><span class="na">createFromElement</span><span class="p">(</span><span class="n">HqlSqlWalker</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">324</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">antlr</span><span class="p">.</span><span class="na">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">fromElement</span><span class="p">(</span><span class="n">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">3420</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">antlr</span><span class="p">.</span><span class="na">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">fromElementList</span><span class="p">(</span><span class="n">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">3309</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">antlr</span><span class="p">.</span><span class="na">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">fromClause</span><span class="p">(</span><span class="n">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">706</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">antlr</span><span class="p">.</span><span class="na">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">query</span><span class="p">(</span><span class="n">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">562</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">antlr</span><span class="p">.</span><span class="na">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">selectStatement</span><span class="p">(</span><span class="n">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">299</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">antlr</span><span class="p">.</span><span class="na">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">statement</span><span class="p">(</span><span class="n">HqlSqlBaseWalker</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">247</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">QueryTranslatorImpl</span><span class="p">.</span><span class="na">analyze</span><span class="p">(</span><span class="n">QueryTranslatorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">248</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">QueryTranslatorImpl</span><span class="p">.</span><span class="na">doCompile</span><span class="p">(</span><span class="n">QueryTranslatorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">183</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">hql</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">ast</span><span class="p">.</span><span class="na">QueryTranslatorImpl</span><span class="p">.</span><span class="na">compile</span><span class="p">(</span><span class="n">QueryTranslatorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">136</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">engine</span><span class="p">.</span><span class="na">query</span><span class="p">.</span><span class="na">spi</span><span class="p">.</span><span class="na">HQLQueryPlan</span><span class="p">.</span><span class="o"><</span><span class="n">init</span><span class="o">></span><span class="p">(</span><span class="n">HQLQueryPlan</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">105</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">engine</span><span class="p">.</span><span class="na">query</span><span class="p">.</span><span class="na">spi</span><span class="p">.</span><span class="na">HQLQueryPlan</span><span class="p">.</span><span class="o"><</span><span class="n">init</span><span class="o">></span><span class="p">;(</span><span class="n">HQLQueryPlan</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">80</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">engine</span><span class="p">.</span><span class="na">query</span><span class="p">.</span><span class="na">spi</span><span class="p">.</span><span class="na">QueryPlanCache</span><span class="p">.</span><span class="na">getHQLQueryPlan</span><span class="p">(</span><span class="n">QueryPlanCache</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">168</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">AbstractSessionImpl</span><span class="p">.</span><span class="na">getHQLQueryPlan</span><span class="p">(</span><span class="n">AbstractSessionImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">221</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">AbstractSessionImpl</span><span class="p">.</span><span class="na">createQuery</span><span class="p">(</span><span class="n">AbstractSessionImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">199</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">SessionImpl</span><span class="p">.</span><span class="na">createQuery</span><span class="p">(</span><span class="n">SessionImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">1740</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">bm</span><span class="p">.</span><span class="na">alarms</span><span class="p">.</span><span class="na">AlarmDao</span><span class="p">.</span><span class="na">findAll</span><span class="p">(</span><span class="n">AlarmDao</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">49</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke0</span><span class="p">(</span><span class="n">Native</span><span class="w"> </span><span class="n">Method</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">57</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">43</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">Method</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">Method</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">601</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">aop</span><span class="p">.</span><span class="na">support</span><span class="p">.</span><span class="na">AopUtils</span><span class="p">.</span><span class="na">invokeJoinpointUsingReflection</span><span class="p">(</span><span class="n">AopUtils</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">317</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">aop</span><span class="p">.</span><span class="na">framework</span><span class="p">.</span><span class="na">ReflectiveMethodInvocation</span><span class="p">.</span><span class="na">invokeJoinpoint</span><span class="p">(</span><span class="n">ReflectiveMethodInvocation</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">183</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">aop</span><span class="p">.</span><span class="na">framework</span><span class="p">.</span><span class="na">ReflectiveMethodInvocation</span><span class="p">.</span><span class="na">proceed</span><span class="p">(</span><span class="n">ReflectiveMethodInvocation</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">150</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">transaction</span><span class="p">.</span><span class="na">interceptor</span><span class="p">.</span><span class="na">TransactionInterceptor$1</span><span class="p">.</span><span class="na">proceedWithInvocation</span><span class="p">(</span><span class="n">TransactionInterceptor</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">96</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">transaction</span><span class="p">.</span><span class="na">interceptor</span><span class="p">.</span><span class="na">TransactionAspectSupport</span><span class="p">.</span><span class="na">invokeWithinTransaction</span><span class="p">(</span><span class="n">TransactionAspectSupport</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">260</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">transaction</span><span class="p">.</span><span class="na">interceptor</span><span class="p">.</span><span class="na">TransactionInterceptor</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">TransactionInterceptor</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">94</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">aop</span><span class="p">.</span><span class="na">framework</span><span class="p">.</span><span class="na">ReflectiveMethodInvocation</span><span class="p">.</span><span class="na">proceed</span><span class="p">(</span><span class="n">ReflectiveMethodInvocation</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">172</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">aop</span><span class="p">.</span><span class="na">framework</span><span class="p">.</span><span class="na">JdkDynamicAopProxy</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">JdkDynamicAopProxy</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">204</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">$Proxy14</span><span class="p">.</span><span class="na">findAll</span><span class="p">(</span><span class="n">Unknown</span><span class="w"> </span><span class="n">Source</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">bm</span><span class="p">.</span><span class="na">alarms</span><span class="p">.</span><span class="na">AlarmBo</span><span class="p">.</span><span class="na">findAll</span><span class="p">(</span><span class="n">AlarmBo</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">35</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">bm</span><span class="p">.</span><span class="na">Evaluate</span><span class="p">.</span><span class="na">run_db</span><span class="p">(</span><span class="n">Evaluate</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">55</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">bm</span><span class="p">.</span><span class="na">App</span><span class="p">.</span><span class="na">main</span><span class="p">(</span><span class="n">App</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">22</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>The data access object that actually throwed this exception is</p>
<div class="highlight"><pre><span></span><code><span class="kd">public</span><span class="w"> </span><span class="n">List</span><span class="o"><</span><span class="n">Foo</span><span class="o">></span><span class="w"> </span><span class="nf">findAll</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">sessionFactory</span><span class="p">.</span><span class="na">getCurrentSession</span><span class="p">().</span><span class="na">createQuery</span><span class="p">(</span><span class="s">"from foo"</span><span class="p">).</span><span class="na">list</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>The mistake I did in the above snippet is that I have used the table name <code>foo</code> inside createQuery. Instead, I got to use <code>Foo</code>, the actual class name.</p>
<div class="highlight"><pre><span></span><code><span class="kd">public</span><span class="w"> </span><span class="n">List</span><span class="o"><</span><span class="n">Foo</span><span class="o">></span><span class="w"> </span><span class="nf">findAll</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">sessionFactory</span><span class="p">.</span><span class="na">getCurrentSession</span><span class="p">().</span><span class="na">createQuery</span><span class="p">(</span><span class="s">"from Foo"</span><span class="p">).</span><span class="na">list</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Boom!!! Exception resolved.</p>Automatically generating slug url in django2013-04-03T15:30:00+05:302013-04-03T15:30:00+05:30ArunDhaJtag:www.arundhaj.com,2013-04-03:/blog/automatically-generating-slug-url.html<p>Auto generating a random clean URL (slug URL) for django web application. In URL <code>www.website.com/app/d4Ers</code> the last part is auto generated random string.</p>
<p>We need to first install the following package</p>
<div class="highlight"><pre><span></span><code>$ sudo pip install django-autoslug
</code></pre></div>
<p>then import the package</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">autoslug.fields</span> <span class="kn">import</span> <span class="n">AutoSlugField</span>
</code></pre></div>
<p>In the …</p><p>Auto generating a random clean URL (slug URL) for django web application. In URL <code>www.website.com/app/d4Ers</code> the last part is auto generated random string.</p>
<p>We need to first install the following package</p>
<div class="highlight"><pre><span></span><code>$ sudo pip install django-autoslug
</code></pre></div>
<p>then import the package</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">autoslug.fields</span> <span class="kn">import</span> <span class="n">AutoSlugField</span>
</code></pre></div>
<p>In the model class add a field</p>
<div class="highlight"><pre><span></span><code><span class="n">url_ref</span> <span class="o">=</span> <span class="n">AutoSlugField</span><span class="p">(</span><span class="n">slugify</span><span class="o">=</span><span class="k">lambda</span> <span class="n">instance</span><span class="p">:</span> <span class="s1">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">sample</span><span class="p">(</span><span class="n">string</span><span class="o">.</span><span class="n">uppercase</span> <span class="o">+</span> <span class="n">string</span><span class="o">.</span><span class="n">lowercase</span> <span class="o">+</span> <span class="n">string</span><span class="o">.</span><span class="n">digits</span><span class="p">,</span> <span class="mi">5</span><span class="p">)),</span> <span class="n">unique</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</code></pre></div>
<p>Hope this helps.</p>gitignore for Python projects2013-02-16T11:00:00+05:302013-02-16T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-02-16:/blog/gitignore-for-python-projects.html<p><a href="http://git-scm.com/">GIT</a> repo's gitignore file for the <a href="http://python.org/">python</a> <a href="https://www.djangoproject.com/">django</a> projects I use. Though it is not extensive, I would update as I encounter new formats.</p>
<div class="highlight"><pre><span></span><code>*.py<span class="o">[</span>cod<span class="o">]</span>
<span class="c1"># Packages</span>
*.egg
*.egg-info
dist
build
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg
lib
lib64
__pycache__
<span class="c1"># Installer logs</span>
pip-log.txt
<span class="c1"># Unit test / coverage reports …</span></code></pre></div><p><a href="http://git-scm.com/">GIT</a> repo's gitignore file for the <a href="http://python.org/">python</a> <a href="https://www.djangoproject.com/">django</a> projects I use. Though it is not extensive, I would update as I encounter new formats.</p>
<div class="highlight"><pre><span></span><code>*.py<span class="o">[</span>cod<span class="o">]</span>
<span class="c1"># Packages</span>
*.egg
*.egg-info
dist
build
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg
lib
lib64
__pycache__
<span class="c1"># Installer logs</span>
pip-log.txt
<span class="c1"># Unit test / coverage reports</span>
.coverage
.tox
nosetests.xml
<span class="c1"># Eclipse pydev</span>
.project
.pydevproject
</code></pre></div>Connecting internet with USB modem - Raspberry Pi2013-02-09T11:00:00+05:302013-02-09T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-02-09:/blog/connecting-internet-with-usb-modem.html<p>After some exploration, I could successfully connet to the internet with Tata Photon Plus from my Pi. Though with Raspbian, it is not as straight forward as it is with any other Linux varient.</p>
<p>There are many options available for connecting to the internet, based on the available infrastructure. In …</p><p>After some exploration, I could successfully connet to the internet with Tata Photon Plus from my Pi. Though with Raspbian, it is not as straight forward as it is with any other Linux varient.</p>
<p>There are many options available for connecting to the internet, based on the available infrastructure. In my case, I have a Tata Photon Plus (Olive VME102) USB modem which I use it on my laptop directly. With establishing <a href="http://arundhaj.com/blog/raspberry-pi-peer-to-peer.html">peer-to-peer connetion</a>, I first tried to share the internet from my Windows 7 laptop to the Pi. I still didn't found a way out and working on it. Later, decided to first bring my Pi on the internet then started to do this. Some of the steps requires external modules that has to be downloaded. So, during these setup process, the Pi has to be in the internet. I connected my Pi in my office intranet for initial configuration. </p>
<p>Your <code>/etc/network/interfaces</code> file should contain the configuration for connecting to the office intranet.</p>
<div class="highlight"><pre><span></span><code>auto eth0
iface eth0 inet dhcp
</code></pre></div>
<p>When the USB modem is connected for the first time it would be detected as a storage device and would be using usb-storage driver. My device was assigned to <code>/dev/sr0</code> file, which is a storage file. Check it with </p>
<div class="highlight"><pre><span></span><code>$ ls -l /dev/disk/by-id/usb*
</code></pre></div>
<p>Load the device's specific module with <code>modprobe usbserial vendor=0x22F4 product=0x0021</code>. Then check which driver has been loaded for the device with <code>usb-devices</code> command. Mostly it would be showing as usb-storage. Now <code>usb_modeswitch</code> has to be installed to change it to a serial device. Once installed, it will automatically change. Seems there are few unsupported devices, in which case, it has to be done manually. Upon success, the device would be reassiged to <code>/dev/ttyUSB0</code>. Confirm it with <code>dmesg</code>.</p>
<p>Further, install wvdial and your <code>/etc/wvdial.conf</code> file should look like this.</p>
<div class="highlight"><pre><span></span><code><span class="nv">Init</span> <span class="o">=</span> ATZ
<span class="nv">Init</span> <span class="o">=</span> ATQ0 V1 E1 <span class="nv">S0</span><span class="o">=</span><span class="m">0</span> <span class="p">&</span>C1 <span class="p">&</span>D2 +FCLASS<span class="o">=</span><span class="m">0</span>
Modem <span class="nv">Type</span> <span class="o">=</span> USB Modem
<span class="nv">Baud</span> <span class="o">=</span> <span class="m">115200</span>
<span class="nv">Modem</span> <span class="o">=</span> /dev/ttyUSB0
<span class="nv">Phone</span> <span class="o">=</span> <span class="c1">#777</span>
<span class="nv">Username</span> <span class="o">=</span> internet
<span class="nv">Password</span> <span class="o">=</span> internet
Stupid <span class="nv">Mode</span> <span class="o">=</span> <span class="m">1</span>
</code></pre></div>
<p>Then start dialing and enjoy browsing!!!</p>
<div class="highlight"><pre><span></span><code>$ sudo wvdial
</code></pre></div>Peer to Peer networking with Raspberry Pi2013-01-24T22:00:00+05:302013-01-24T22:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-01-24:/blog/peer-to-peer-networking-with-raspberry-pi.html<p>In this post, I'll explain how to setup a peer-to-peer local network between Windows 7 machine and the <a href="http://www.raspberrypi.org">Raspberry Pi</a>. The raspbian version I used is <a href="http://downloads.raspberrypi.org/images/raspbian/2012-12-16-wheezy-raspbian/2012-12-16-wheezy-raspbian.zip">2012-12-16-wheezy-raspbian</a>.</p>
<p>For networking we require an ethernet crossover cable. (Some say that pi does automatic crossover and hence straight cable is sufficient, however, I …</p><p>In this post, I'll explain how to setup a peer-to-peer local network between Windows 7 machine and the <a href="http://www.raspberrypi.org">Raspberry Pi</a>. The raspbian version I used is <a href="http://downloads.raspberrypi.org/images/raspbian/2012-12-16-wheezy-raspbian/2012-12-16-wheezy-raspbian.zip">2012-12-16-wheezy-raspbian</a>.</p>
<p>For networking we require an ethernet crossover cable. (Some say that pi does automatic crossover and hence straight cable is sufficient, however, I haven't checked it yet.)</p>
<p>First connect both the machines with the ethernet cable, then make a note of windows <code>IP address</code> and <code>Subnet Mask</code>. This is essential, because, the static IP that we assign to the pi should be in the same subnet. Though this is obvious, most of the time people forget it (as I did) and assign IP under different subnet; which will never work without a switch, hub or router. Alternatively, both windows IP and pi's IP could be manually configured with same subnet. However since windows assign an IP automatically, it is sufficient to configure only the pi.</p>
<p>Fine, in my case, following is my windows IP configuration.</p>
<div class="highlight"><pre><span></span><code>Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
Autoconfiguration IPv4 Address. . : <span class="m">169</span>.254.253.161
Subnet Mask . . . . . . . . . . . : <span class="m">255</span>.255.0.0
Default Gateway . . . . . . . . . :
</code></pre></div>
<p>So, accordingly configure pi. Edit the interface file <code>/etc/network/interfaces</code> and make sure the following content is fed.</p>
<div class="highlight"><pre><span></span><code>auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address <span class="m">169</span>.254.253.162
netmask <span class="m">255</span>.255.0.0
</code></pre></div>
<p>Restart the network <code>$ sudo /etc/init.d/networking restart</code> and ensure the settings are reflected. Also, check if the SSH server is enabled, else enable it.</p>
<p>Thats it. On windows use <a href="http://www.putty.org/">PuTTY</a> to establish the SSH connection with pi.</p>
<p>I'm now working on sharing internet from Windows to Pi.</p>J2EE server for Webfaction2013-01-10T10:00:00+05:302013-01-10T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-01-10:/blog/j2ee-server-for-webfaction.html<p>Webfaction doesn't have j2ee web servers by default. Users have to install it manually, by creating a "custom application listening on port".</p>
<p>I first started with <a href="http://tomcat.apache.org/">Apache Tomcat</a>, later ended with high memory usage than permitted for my account. My account's limit is 256MB RAM, however Tomcat was consuming around …</p><p>Webfaction doesn't have j2ee web servers by default. Users have to install it manually, by creating a "custom application listening on port".</p>
<p>I first started with <a href="http://tomcat.apache.org/">Apache Tomcat</a>, later ended with high memory usage than permitted for my account. My account's limit is 256MB RAM, however Tomcat was consuming around 500MB. Due to this reason, webfaction automatically terminates this high memory consuming process. I tried setting <code>CATALINA_OPTS</code> with heap size restriction with no success.</p>
<div class="highlight"><pre><span></span><code>$ ps -u arundhaj -o rss,size,etime,pid,command
RSS SZ ELAPSED PID COMMAND
<span class="m">524668</span> <span class="m">6118968</span> <span class="m">00</span>:32 <span class="m">3630</span> /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java -Djava.util.logging.config.file<span class="o">=</span>/home/arund
<span class="m">1128</span> <span class="m">1060</span> <span class="m">00</span>:00 <span class="m">4267</span> ps -u arundhaj -o rss,size,etime,pid,command
<span class="m">1740</span> <span class="m">792</span> <span class="m">11</span>:16 <span class="m">23052</span> sshd: arundhaj@pts/16
<span class="m">1976</span> <span class="m">560</span> <span class="m">11</span>:15 <span class="m">23073</span> -bash
</code></pre></div>
<p>Later I deployed <a href="http://jetty.codehaus.org/jetty/">Jetty</a>. So far, I've hosted only one web application and the memory footage is considerably less.</p>
<div class="highlight"><pre><span></span><code>$ ps -u arundhaj -o rss,size,etime,pid,command
RSS SZ ELAPSED PID COMMAND
<span class="m">1064</span> <span class="m">1056</span> <span class="m">00</span>:01 <span class="m">18762</span> ps -u arundhaj -o rss,size,etime,pid,command
<span class="m">187588</span> <span class="m">2557360</span> <span class="m">10</span>:57:50 <span class="m">21485</span> /usr/bin/java -Djetty.home<span class="o">=</span>/home/arundhaj/webapps/jetty -Djava.io.tmpdir<span class="o">=</span>/tmp -server -Xmx
<span class="m">1740</span> <span class="m">792</span> <span class="m">14</span>:27 <span class="m">31562</span> sshd: arundhaj@pts/8
<span class="m">1892</span> <span class="m">424</span> <span class="m">14</span>:26 <span class="m">31587</span> -bash
</code></pre></div>
<p>I still use the following option in jetty server startup, so that, I don't exceed the limit. </p>
<div class="highlight"><pre><span></span><code><span class="nv">JAVA_OPTIONS</span><span class="o">+=(</span><span class="s2">"-server -Xmx256m -Djava.awt.headless=true"</span><span class="o">)</span>
</code></pre></div>
<p>Let us see, how long does this jetty server withstand this memory restriction. </p>
<p>If you know of any server lightweight than Tomcat and Jetty, please let me know.</p>UnsupportedClassVersionError unsupported major.minor version 51.02013-01-08T21:00:00+05:302013-01-08T21:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-01-08:/blog/unsupportedclassversionerror-unsupported-major-minor-version-51-0.html<p>For J2EE webapps, when development environment is different from production environment, deploying could be tedious.</p>
<p>One such issue I encountered is the difference in JDK version. My development machine have latest JDK1.7 whereas my production servers have JDK1.6. When deploying the war on this server, I was getting …</p><p>For J2EE webapps, when development environment is different from production environment, deploying could be tedious.</p>
<p>One such issue I encountered is the difference in JDK version. My development machine have latest JDK1.7 whereas my production servers have JDK1.6. When deploying the war on this server, I was getting the following exception catpured in catalina.out</p>
<div class="highlight"><pre><span></span><code>Caused by: java.lang.UnsupportedClassVersionError: com/arundhaj/controller/TestController : Unsupported major.minor version <span class="m">51</span>.0 <span class="o">(</span>unable to load class com.arundhaj.controller.TestController<span class="o">)</span>
at org.apache.catalina.loader.WebappClassLoader.findClassInternal<span class="o">(</span>WebappClassLoader.java:2908<span class="o">)</span>
at org.apache.catalina.loader.WebappClassLoader.findClass<span class="o">(</span>WebappClassLoader.java:1173<span class="o">)</span>
at org.apache.catalina.loader.WebappClassLoader.loadClass<span class="o">(</span>WebappClassLoader.java:1681<span class="o">)</span>
at org.apache.catalina.loader.WebappClassLoader.loadClass<span class="o">(</span>WebappClassLoader.java:1559<span class="o">)</span>
at org.springframework.util.ClassUtils.forName<span class="o">(</span>ClassUtils.java:258<span class="o">)</span>
at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass<span class="o">(</span>AbstractBeanDefinition.java:417<span class="o">)</span>
at org.springframework.beans.factory.support.AbstractBeanFactory.doResolveBeanClass<span class="o">(</span>AbstractBeanFactory.java:1283<span class="o">)</span>
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass<span class="o">(</span>AbstractBeanFactory.java:1254<span class="o">)</span>
... <span class="m">23</span> more
Jan <span class="m">8</span>, <span class="m">2013</span> <span class="m">9</span>:39:45 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Error listenerStart
</code></pre></div>
<p>To get rid of this, we need to tell eclipse to build the war for production servers' target. I am using maven, hence, in <code>maven-compiler-plugin</code> set target to 1.6 as</p>
<div class="highlight"><pre><span></span><code><span class="nt"><configuration></span>
<span class="nt"><source></span>1.6<span class="nt"></source></span>
<span class="nt"><target></span>1.6<span class="nt"></target></span>
<span class="nt"></configuration></span>
</code></pre></div>Hbase row filter with regex2013-01-06T18:00:00+05:302013-01-06T18:00:00+05:30ArunDhaJtag:www.arundhaj.com,2013-01-06:/blog/hbase-row-filter-with-regex.html<p>Querying Hbase rows based on RegEx with spring-data-hadoop framework; following code snippet would help.</p>
<p>To keep it simple, I query the table to get only those row keys that match my INPUT. In this case the row key structure is 10 digit followed by a string. </p>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">String</span><span class="p">.</span><span class="na">format …</span></code></pre></div><p>Querying Hbase rows based on RegEx with spring-data-hadoop framework; following code snippet would help.</p>
<p>To keep it simple, I query the table to get only those row keys that match my INPUT. In this case the row key structure is 10 digit followed by a string. </p>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">String</span><span class="p">.</span><span class="na">format</span><span class="p">(</span><span class="s">"^\\d{10}%s$"</span><span class="p">,</span><span class="w"> </span><span class="n">INPUT</span><span class="p">);</span><span class="w"></span>
<span class="n">RegexStringComparator</span><span class="w"> </span><span class="n">keyRegEx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">RegexStringComparator</span><span class="p">(</span><span class="n">name</span><span class="p">);</span><span class="w"></span>
<span class="n">RowFilter</span><span class="w"> </span><span class="n">rowFilter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">RowFilter</span><span class="p">(</span><span class="n">CompareOp</span><span class="p">.</span><span class="na">EQUAL</span><span class="p">,</span><span class="w"> </span><span class="n">keyRegEx</span><span class="p">);</span><span class="w"></span>
<span class="n">Scan</span><span class="w"> </span><span class="n">rowScan</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">Scan</span><span class="p">();</span><span class="w"></span>
<span class="n">rowScan</span><span class="p">.</span><span class="na">setFilter</span><span class="p">(</span><span class="n">rowFilter</span><span class="p">);</span><span class="w"></span>
<span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="n">children</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">getHbaseTemplate</span><span class="p">().</span><span class="na">find</span><span class="p">(</span><span class="n">TABLE_NAME</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">rowScan</span><span class="p">,</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">ResultsExtractor</span><span class="o"><</span><span class="n">List</span><span class="o">></span><span class="n">String</span><span class="o">>></span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nd">@Override</span><span class="w"></span>
<span class="w"> </span><span class="kd">public</span><span class="w"> </span><span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="nf">extractData</span><span class="p">(</span><span class="n">ResultScanner</span><span class="w"> </span><span class="n">rs</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">Exception</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="w"> </span><span class="n">children</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">ArrayList</span><span class="o"><</span><span class="n">String</span><span class="o">></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">Result</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rs</span><span class="p">.</span><span class="na">next</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">String</span><span class="w"> </span><span class="n">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Bytes</span><span class="p">.</span><span class="na">toString</span><span class="p">(</span><span class="n">result</span><span class="p">.</span><span class="na">getRow</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">children</span><span class="p">.</span><span class="na">add</span><span class="p">(</span><span class="n">key</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>For reference visit the project <a href="https://github.com/arundhaj/springdatahadoop">github</a></p>Spring data hadoop webapp2012-12-11T15:00:00+05:302012-12-11T15:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-12-11:/blog/spring-data-hadoop-webapp.html<p>When I got started to try <a href="http://www.springsource.org/spring-data/hadoop">spring-data-hadoop</a> as a webapp, I found no samples online. All tutorials were either with configurations or as a standalone application. With a bit of experiments and exploration I managed to create a simple webapp and hosted it on my local tomcat server. I've uploaded …</p><p>When I got started to try <a href="http://www.springsource.org/spring-data/hadoop">spring-data-hadoop</a> as a webapp, I found no samples online. All tutorials were either with configurations or as a standalone application. With a bit of experiments and exploration I managed to create a simple webapp and hosted it on my local tomcat server. I've uploaded the project on <a href="https://github.com/arundhaj/springdatahadoop">github</a> expecting that it might help someone. </p>
<p>However, still I'm facing the below exception when running a mapreduce job from the Servlet controller.</p>
<div class="highlight"><pre><span></span><code><span class="n">Dec</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">2012</span><span class="w"> </span><span class="mi">3</span><span class="p">:</span><span class="mi">58</span><span class="p">:</span><span class="mi">12</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">NativeCodeLoader</span><span class="w"> </span><span class="o"><</span><span class="n">clinit</span><span class="o">></span><span class="w"></span>
<span class="nl">WARNING</span><span class="p">:</span><span class="w"> </span><span class="n">Unable</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">load</span><span class="w"> </span><span class="kd">native</span><span class="o">-</span><span class="n">hadoop</span><span class="w"> </span><span class="n">library</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="n">platform</span><span class="p">...</span><span class="w"> </span><span class="n">using</span><span class="w"> </span><span class="n">builtin</span><span class="o">-</span><span class="n">java</span><span class="w"> </span><span class="n">classes</span><span class="w"> </span><span class="n">where</span><span class="w"> </span><span class="n">applicable</span><span class="w"></span>
<span class="n">Dec</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">2012</span><span class="w"> </span><span class="mi">3</span><span class="p">:</span><span class="mi">58</span><span class="p">:</span><span class="mi">12</span><span class="w"> </span><span class="n">PM</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">security</span><span class="p">.</span><span class="na">UserGroupInformation</span><span class="w"> </span><span class="n">doAs</span><span class="w"></span>
<span class="nl">SEVERE</span><span class="p">:</span><span class="w"> </span><span class="n">PriviledgedActionException</span><span class="w"> </span><span class="n">as</span><span class="p">:</span><span class="n">tomcat7</span><span class="w"> </span><span class="n">cause</span><span class="p">:</span><span class="n">java</span><span class="p">.</span><span class="na">io</span><span class="p">.</span><span class="na">IOException</span><span class="p">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">set</span><span class="w"> </span><span class="n">permissions</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">path</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">hadoop_ws</span><span class="o">/</span><span class="n">mapred</span><span class="o">/</span><span class="n">staging</span><span class="o">/</span><span class="n">tomcat7</span><span class="o">-</span><span class="mi">58714176</span><span class="o">/</span><span class="p">.</span><span class="na">staging</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="mo">0700</span><span class="w"></span>
<span class="n">java</span><span class="p">.</span><span class="na">io</span><span class="p">.</span><span class="na">IOException</span><span class="p">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">set</span><span class="w"> </span><span class="n">permissions</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">path</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">hadoop_ws</span><span class="o">/</span><span class="n">mapred</span><span class="o">/</span><span class="n">staging</span><span class="o">/</span><span class="n">tomcat7</span><span class="o">-</span><span class="mi">58714176</span><span class="o">/</span><span class="p">.</span><span class="na">staging</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="mo">0700</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">fs</span><span class="p">.</span><span class="na">FileUtil</span><span class="p">.</span><span class="na">checkReturnValue</span><span class="p">(</span><span class="n">FileUtil</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">689</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">fs</span><span class="p">.</span><span class="na">FileUtil</span><span class="p">.</span><span class="na">setPermission</span><span class="p">(</span><span class="n">FileUtil</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">662</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">fs</span><span class="p">.</span><span class="na">RawLocalFileSystem</span><span class="p">.</span><span class="na">setPermission</span><span class="p">(</span><span class="n">RawLocalFileSystem</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">509</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">fs</span><span class="p">.</span><span class="na">RawLocalFileSystem</span><span class="p">.</span><span class="na">mkdirs</span><span class="p">(</span><span class="n">RawLocalFileSystem</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">344</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">fs</span><span class="p">.</span><span class="na">FilterFileSystem</span><span class="p">.</span><span class="na">mkdirs</span><span class="p">(</span><span class="n">FilterFileSystem</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">189</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">mapreduce</span><span class="p">.</span><span class="na">JobSubmissionFiles</span><span class="p">.</span><span class="na">getStagingDir</span><span class="p">(</span><span class="n">JobSubmissionFiles</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">116</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">mapred</span><span class="p">.</span><span class="na">JobClient$2</span><span class="p">.</span><span class="na">run</span><span class="p">(</span><span class="n">JobClient</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">856</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">mapred</span><span class="p">.</span><span class="na">JobClient$2</span><span class="p">.</span><span class="na">run</span><span class="p">(</span><span class="n">JobClient</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">850</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">security</span><span class="p">.</span><span class="na">AccessController</span><span class="p">.</span><span class="na">doPrivileged</span><span class="p">(</span><span class="n">Native</span><span class="w"> </span><span class="n">Method</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">javax</span><span class="p">.</span><span class="na">security</span><span class="p">.</span><span class="na">auth</span><span class="p">.</span><span class="na">Subject</span><span class="p">.</span><span class="na">doAs</span><span class="p">(</span><span class="n">Subject</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">415</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">security</span><span class="p">.</span><span class="na">UserGroupInformation</span><span class="p">.</span><span class="na">doAs</span><span class="p">(</span><span class="n">UserGroupInformation</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">1121</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">mapred</span><span class="p">.</span><span class="na">JobClient</span><span class="p">.</span><span class="na">submitJobInternal</span><span class="p">(</span><span class="n">JobClient</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">850</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">mapreduce</span><span class="p">.</span><span class="na">Job</span><span class="p">.</span><span class="na">submit</span><span class="p">(</span><span class="n">Job</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">500</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">hadoop</span><span class="p">.</span><span class="na">mapreduce</span><span class="p">.</span><span class="na">Job</span><span class="p">.</span><span class="na">waitForCompletion</span><span class="p">(</span><span class="n">Job</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">530</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">controller</span><span class="p">.</span><span class="na">HelloController</span><span class="p">.</span><span class="na">sayHello</span><span class="p">(</span><span class="n">HelloController</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">55</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke0</span><span class="p">(</span><span class="n">Native</span><span class="w"> </span><span class="n">Method</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">57</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">43</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">Method</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">Method</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">601</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">support</span><span class="p">.</span><span class="na">InvocableHandlerMethod</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">InvocableHandlerMethod</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">219</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">support</span><span class="p">.</span><span class="na">InvocableHandlerMethod</span><span class="p">.</span><span class="na">invokeForRequest</span><span class="p">(</span><span class="n">InvocableHandlerMethod</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">132</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">annotation</span><span class="p">.</span><span class="na">ServletInvocableHandlerMethod</span><span class="p">.</span><span class="na">invokeAndHandle</span><span class="p">(</span><span class="n">ServletInvocableHandlerMethod</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">100</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">annotation</span><span class="p">.</span><span class="na">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">invokeHandlerMethod</span><span class="p">(</span><span class="n">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">604</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">annotation</span><span class="p">.</span><span class="na">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">handleInternal</span><span class="p">(</span><span class="n">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">565</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">AbstractHandlerMethodAdapter</span><span class="p">.</span><span class="na">handle</span><span class="p">(</span><span class="n">AbstractHandlerMethodAdapter</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">80</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">DispatcherServlet</span><span class="p">.</span><span class="na">doDispatch</span><span class="p">(</span><span class="n">DispatcherServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">923</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">DispatcherServlet</span><span class="p">.</span><span class="na">doService</span><span class="p">(</span><span class="n">DispatcherServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">852</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">FrameworkServlet</span><span class="p">.</span><span class="na">processRequest</span><span class="p">(</span><span class="n">FrameworkServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">882</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">FrameworkServlet</span><span class="p">.</span><span class="na">doGet</span><span class="p">(</span><span class="n">FrameworkServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">778</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">javax</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">http</span><span class="p">.</span><span class="na">HttpServlet</span><span class="p">.</span><span class="na">service</span><span class="p">(</span><span class="n">HttpServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">621</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">javax</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">http</span><span class="p">.</span><span class="na">HttpServlet</span><span class="p">.</span><span class="na">service</span><span class="p">(</span><span class="n">HttpServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">722</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">core</span><span class="p">.</span><span class="na">ApplicationFilterChain</span><span class="p">.</span><span class="na">internalDoFilter</span><span class="p">(</span><span class="n">ApplicationFilterChain</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">305</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">core</span><span class="p">.</span><span class="na">ApplicationFilterChain</span><span class="p">.</span><span class="na">doFilter</span><span class="p">(</span><span class="n">ApplicationFilterChain</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">210</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">core</span><span class="p">.</span><span class="na">StandardWrapperValve</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">StandardWrapperValve</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">224</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">core</span><span class="p">.</span><span class="na">StandardContextValve</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">StandardContextValve</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">169</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">authenticator</span><span class="p">.</span><span class="na">AuthenticatorBase</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">AuthenticatorBase</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">472</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">core</span><span class="p">.</span><span class="na">StandardHostValve</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">StandardHostValve</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">168</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">valves</span><span class="p">.</span><span class="na">ErrorReportValve</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">ErrorReportValve</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">98</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">valves</span><span class="p">.</span><span class="na">AccessLogValve</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">AccessLogValve</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">927</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">core</span><span class="p">.</span><span class="na">StandardEngineValve</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">StandardEngineValve</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">118</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">catalina</span><span class="p">.</span><span class="na">connector</span><span class="p">.</span><span class="na">CoyoteAdapter</span><span class="p">.</span><span class="na">service</span><span class="p">(</span><span class="n">CoyoteAdapter</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">407</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">coyote</span><span class="p">.</span><span class="na">http11</span><span class="p">.</span><span class="na">AbstractHttp11Processor</span><span class="p">.</span><span class="na">process</span><span class="p">(</span><span class="n">AbstractHttp11Processor</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">987</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">coyote</span><span class="p">.</span><span class="na">AbstractProtocol$AbstractConnectionHandler</span><span class="p">.</span><span class="na">process</span><span class="p">(</span><span class="n">AbstractProtocol</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">579</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">apache</span><span class="p">.</span><span class="na">tomcat</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">net</span><span class="p">.</span><span class="na">JIoEndpoint$SocketProcessor</span><span class="p">.</span><span class="na">run</span><span class="p">(</span><span class="n">JIoEndpoint</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">307</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">concurrent</span><span class="p">.</span><span class="na">ThreadPoolExecutor</span><span class="p">.</span><span class="na">runWorker</span><span class="p">(</span><span class="n">ThreadPoolExecutor</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">1110</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">concurrent</span><span class="p">.</span><span class="na">ThreadPoolExecutor$Worker</span><span class="p">.</span><span class="na">run</span><span class="p">(</span><span class="n">ThreadPoolExecutor</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">603</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">Thread</span><span class="p">.</span><span class="na">run</span><span class="p">(</span><span class="n">Thread</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">722</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>After some troubleshooting with no luck, posted the query in springsource <a href="http://forum.springsource.org/showthread.php?133210-PriviledgedActionException-when-running-mapreduce-job-from-servlet">community</a>. If any do know the solution please let me know.</p>
<h4>My development environment</h4>
<ul>
<li>Ubuntu 12.04</li>
<li>Tomcat 7</li>
<li>Apache Hadoop 1.0.4</li>
<li>Apache HBase 0.94.2</li>
<li>Spring 3.1.3.RELEASE</li>
<li>spring-data-hadoop 1.0.0.RC1</li>
</ul>HibernateException No Session found for current thread2012-12-08T20:30:00+05:302012-12-08T20:30:00+05:30ArunDhaJtag:www.arundhaj.com,2012-12-08:/blog/hibernateexception-no-session-found.html<p>While programming a web application with Spring and Hibernate, came across <code>HibernateException: No session found for current thread</code>. Apparently found out I was missing <code>@Transaction</code> which was causing trouble.</p>
<div class="highlight"><pre><span></span><code><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">NestedServletException</span><span class="p">:</span><span class="w"> </span><span class="n">Request</span><span class="w"> </span><span class="n">processing</span><span class="w"> </span><span class="n">failed</span><span class="p">;</span><span class="w"> </span><span class="n">nested</span><span class="w"> </span><span class="n">exception</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">HibernateException</span><span class="p">:</span><span class="w"> </span><span class="n">No</span><span class="w"> </span><span class="n">Session</span><span class="w"> </span><span class="n">found</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">current</span><span class="w"> </span><span class="n">thread</span><span class="w"></span>
<span class="w"> </span><span class="n">org …</span></code></pre></div><p>While programming a web application with Spring and Hibernate, came across <code>HibernateException: No session found for current thread</code>. Apparently found out I was missing <code>@Transaction</code> which was causing trouble.</p>
<div class="highlight"><pre><span></span><code><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">util</span><span class="p">.</span><span class="na">NestedServletException</span><span class="p">:</span><span class="w"> </span><span class="n">Request</span><span class="w"> </span><span class="n">processing</span><span class="w"> </span><span class="n">failed</span><span class="p">;</span><span class="w"> </span><span class="n">nested</span><span class="w"> </span><span class="n">exception</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">HibernateException</span><span class="p">:</span><span class="w"> </span><span class="n">No</span><span class="w"> </span><span class="n">Session</span><span class="w"> </span><span class="n">found</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">current</span><span class="w"> </span><span class="n">thread</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">FrameworkServlet</span><span class="p">.</span><span class="na">processRequest</span><span class="p">(</span><span class="n">FrameworkServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">894</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">FrameworkServlet</span><span class="p">.</span><span class="na">doGet</span><span class="p">(</span><span class="n">FrameworkServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">778</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">javax</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">http</span><span class="p">.</span><span class="na">HttpServlet</span><span class="p">.</span><span class="na">service</span><span class="p">(</span><span class="n">HttpServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">621</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">javax</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">http</span><span class="p">.</span><span class="na">HttpServlet</span><span class="p">.</span><span class="na">service</span><span class="p">(</span><span class="n">HttpServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">722</span><span class="p">)</span><span class="w"></span>
<span class="n">root</span><span class="w"> </span><span class="n">cause</span><span class="w"></span>
<span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">HibernateException</span><span class="p">:</span><span class="w"> </span><span class="n">No</span><span class="w"> </span><span class="n">Session</span><span class="w"> </span><span class="n">found</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">current</span><span class="w"> </span><span class="n">thread</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">orm</span><span class="p">.</span><span class="na">hibernate4</span><span class="p">.</span><span class="na">SpringSessionContext</span><span class="p">.</span><span class="na">currentSession</span><span class="p">(</span><span class="n">SpringSessionContext</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">97</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">hibernate</span><span class="p">.</span><span class="na">internal</span><span class="p">.</span><span class="na">SessionFactoryImpl</span><span class="p">.</span><span class="na">getCurrentSession</span><span class="p">(</span><span class="n">SessionFactoryImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">978</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">bm</span><span class="p">.</span><span class="na">bookmark</span><span class="p">.</span><span class="na">BookmarkLinksDaoImpl</span><span class="p">.</span><span class="na">findById</span><span class="p">(</span><span class="n">BookmarkLinksDaoImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">39</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke0</span><span class="p">(</span><span class="n">Native</span><span class="w"> </span><span class="n">Method</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">57</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">43</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">Method</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">Method</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">601</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">aop</span><span class="p">.</span><span class="na">support</span><span class="p">.</span><span class="na">AopUtils</span><span class="p">.</span><span class="na">invokeJoinpointUsingReflection</span><span class="p">(</span><span class="n">AopUtils</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">319</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">aop</span><span class="p">.</span><span class="na">framework</span><span class="p">.</span><span class="na">JdkDynamicAopProxy</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">JdkDynamicAopProxy</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">196</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">$Proxy17</span><span class="p">.</span><span class="na">findById</span><span class="p">(</span><span class="n">Unknown</span><span class="w"> </span><span class="n">Source</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">bm</span><span class="p">.</span><span class="na">bookmark</span><span class="p">.</span><span class="na">BookmarkLinksBoImpl</span><span class="p">.</span><span class="na">findById</span><span class="p">(</span><span class="n">BookmarkLinksBoImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">31</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="na">arundhaj</span><span class="p">.</span><span class="na">bm</span><span class="p">.</span><span class="na">controller</span><span class="p">.</span><span class="na">BookmarkController</span><span class="p">.</span><span class="na">bookmark</span><span class="p">(</span><span class="n">BookmarkController</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">39</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke0</span><span class="p">(</span><span class="n">Native</span><span class="w"> </span><span class="n">Method</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">NativeMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">57</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">43</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">reflect</span><span class="p">.</span><span class="na">Method</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">Method</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">601</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">support</span><span class="p">.</span><span class="na">InvocableHandlerMethod</span><span class="p">.</span><span class="na">invoke</span><span class="p">(</span><span class="n">InvocableHandlerMethod</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">219</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">support</span><span class="p">.</span><span class="na">InvocableHandlerMethod</span><span class="p">.</span><span class="na">invokeForRequest</span><span class="p">(</span><span class="n">InvocableHandlerMethod</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">132</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">annotation</span><span class="p">.</span><span class="na">ServletInvocableHandlerMethod</span><span class="p">.</span><span class="na">invokeAndHandle</span><span class="p">(</span><span class="n">ServletInvocableHandlerMethod</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">100</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">annotation</span><span class="p">.</span><span class="na">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">invokeHandlerMethod</span><span class="p">(</span><span class="n">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">604</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">annotation</span><span class="p">.</span><span class="na">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">handleInternal</span><span class="p">(</span><span class="n">RequestMappingHandlerAdapter</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">565</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">mvc</span><span class="p">.</span><span class="na">method</span><span class="p">.</span><span class="na">AbstractHandlerMethodAdapter</span><span class="p">.</span><span class="na">handle</span><span class="p">(</span><span class="n">AbstractHandlerMethodAdapter</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">80</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">DispatcherServlet</span><span class="p">.</span><span class="na">doDispatch</span><span class="p">(</span><span class="n">DispatcherServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">923</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">DispatcherServlet</span><span class="p">.</span><span class="na">doService</span><span class="p">(</span><span class="n">DispatcherServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">852</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">FrameworkServlet</span><span class="p">.</span><span class="na">processRequest</span><span class="p">(</span><span class="n">FrameworkServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">882</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="na">springframework</span><span class="p">.</span><span class="na">web</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">FrameworkServlet</span><span class="p">.</span><span class="na">doGet</span><span class="p">(</span><span class="n">FrameworkServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">778</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">javax</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">http</span><span class="p">.</span><span class="na">HttpServlet</span><span class="p">.</span><span class="na">service</span><span class="p">(</span><span class="n">HttpServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">621</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">javax</span><span class="p">.</span><span class="na">servlet</span><span class="p">.</span><span class="na">http</span><span class="p">.</span><span class="na">HttpServlet</span><span class="p">.</span><span class="na">service</span><span class="p">(</span><span class="n">HttpServlet</span><span class="p">.</span><span class="na">java</span><span class="p">:</span><span class="mi">722</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>Then defined the transaction beans in <code>applicationContext.xml</code></p>
<div class="highlight"><pre><span></span><code><span class="nt"><bean</span> <span class="na">id=</span><span class="s">"transactionManager"</span>
<span class="na">class=</span><span class="s">"org.springframework.orm.hibernate4.HibernateTransactionManager"</span><span class="nt">></span>
<span class="nt"><property</span> <span class="na">name=</span><span class="s">"sessionFactory"</span> <span class="na">ref=</span><span class="s">"sessionFactory"</span><span class="nt">></property></span>
<span class="nt"></bean></span>
<span class="nt"><tx:annotation-driven</span> <span class="na">transaction-manager=</span><span class="s">"transactionManager"</span><span class="nt">/></span>
</code></pre></div>
<p>Further added the annotation to the method which threw the exception.</p>
<div class="highlight"><pre><span></span><code><span class="nd">@Transactional</span><span class="w"></span>
<span class="kd">public</span><span class="w"> </span><span class="n">List</span><span class="w"> </span><span class="nf">findById</span><span class="p">()</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
</code></pre></div>
<p>Hope this helps.</p>Disable animation and improve Ubuntu performance2012-11-02T11:00:00+05:302012-11-02T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-11-02:/blog/disable-animation-ubuntu.html<p>Today I've installed <em>Ubuntu 12.10</em> (Quantal Quetzal). The system is solid; over few minutes of hands-on, one part that bothered me was the windows animation. As a software developer, I know, how difficult it is to implement animation. However, my machine is not fast enough to catch up with …</p><p>Today I've installed <em>Ubuntu 12.10</em> (Quantal Quetzal). The system is solid; over few minutes of hands-on, one part that bothered me was the windows animation. As a software developer, I know, how difficult it is to implement animation. However, my machine is not fast enough to catch up with this, so, I thought to get rid of it.</p>
<p>This is how to do it:</p>
<p>First, install <em>CompizConfig Settings Manager</em>.</p>
<div class="highlight"><pre><span></span><code>$ sudo apt-get install compizconfig-settings-manager
</code></pre></div>
<p>Then start the application. In Categories section, uncheck <em>Animations</em> and <em>Fading Windows</em>.</p>
<p>This would not make a vast improvement, however better. Hope this helps.</p>Moving folders between SVN repos2012-10-26T11:00:00+05:302012-10-26T11:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-10-26:/blog/moving-folders-between-svn-repos.html<p>To move a folder or document between SVN repositories by retaining history, following steps would help.</p>
<h4>Step I</h4>
<p>Dump the entire svn repository to a dump file with <a href="http://svnbook.red-bean.com/en/1.7/svn.ref.svnadmin.c.dump.html">svnadmin dump</a>. If the repository is too big to dump, better pipe the next two commands. Command here is split in the …</p><p>To move a folder or document between SVN repositories by retaining history, following steps would help.</p>
<h4>Step I</h4>
<p>Dump the entire svn repository to a dump file with <a href="http://svnbook.red-bean.com/en/1.7/svn.ref.svnadmin.c.dump.html">svnadmin dump</a>. If the repository is too big to dump, better pipe the next two commands. Command here is split in the interest of easy understanding.</p>
<div class="highlight"><pre><span></span><code>$ svnadmin dump D:<span class="se">\s</span>vn<span class="se">\o</span>ld_repos > .<span class="se">\r</span>epository.dump
</code></pre></div>
<h4>Step II</h4>
<p>Once the repository is dumped, the specific file or folder has to be filtered with <a href="http://svnbook.red-bean.com/en/1.7/svn.ref.svndumpfilter.commands.c.include.html">svndumpfilter include</a> and a new dump has to be created. <code>include</code> is used directly specify the path filter, in case if there are many filter paths, create a file with all paths each in a single line and use <code>-target FILENAME</code>.</p>
<div class="highlight"><pre><span></span><code>$ svndumpfilter include path/to/docs --drop-empty-revs --renumber-revs --preserve-revprops < .<span class="se">\r</span>epository.dump > .<span class="se">\d</span>ocs_only.dump
</code></pre></div>
<h4>Step III</h4>
<p>Now the docs_only.dump has only documents. The next step is to load it to new repository with <a href="http://svnbook.red-bean.com/en/1.7/svn.ref.svnadmin.c.load.html">svnadmin load</a>.</p>
<div class="highlight"><pre><span></span><code>$ svnadmin load D:<span class="se">\s</span>vn<span class="se">\n</span>ew_repos < ./docs_only.dump
</code></pre></div>
<p>While loading, the below mentioned error may occur. </p>
<div class="highlight"><pre><span></span><code><span class="o"><<<</span> Started new transaction, based on original revision <span class="m">1</span>
* adding path : path/to/docs ...svnadmin: E160013: File not found: transaction <span class="s1">'91-2k'</span>, path <span class="s1">'path/to/docs'</span>
</code></pre></div>
<p>It is because, the path is not available in the new repository. In this example <code>path/to</code> doesn't exist. So this path has to be created before loading.</p>getGeneratedKeys with PostgreSQL2012-10-12T17:00:00+05:302012-10-12T17:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-10-12:/blog/getGeneratedKeys-with-postgresql.html<p>In a complex SQL transaction where the tables are normalized and split, it becomes necessary to put primary key of first tables' insert as a foreign key of subsequent tables' inserts. Though <code>java.sql.PreparedStatement</code> have <code>getGeneratedKeys</code>, it works pretty different for PostgreSQL than MySQL.</p>
<p>The following code block shows …</p><p>In a complex SQL transaction where the tables are normalized and split, it becomes necessary to put primary key of first tables' insert as a foreign key of subsequent tables' inserts. Though <code>java.sql.PreparedStatement</code> have <code>getGeneratedKeys</code>, it works pretty different for PostgreSQL than MySQL.</p>
<p>The following code block shows the addition of <code>Statement.RETURN_GENERATED_KEYS</code> which is essential for PostgreSQL</p>
<div class="highlight"><pre><span></span><code><span class="n">String</span><span class="w"> </span><span class="n">insertSql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"INSERT INTO table1 (c1, c2, c3) VALUES (?, ?, ?)"</span><span class="p">;</span><span class="w"></span>
<span class="n">PreparedStatement</span><span class="w"> </span><span class="n">insertPs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">conn</span><span class="p">.</span><span class="na">prepareStatement</span><span class="p">(</span><span class="n">insertSql</span><span class="p">,</span><span class="w"> </span><span class="n">Statement</span><span class="p">.</span><span class="na">RETURN_GENERATED_KEYS</span><span class="p">);</span><span class="w"></span>
<span class="n">insertPs</span><span class="p">.</span><span class="na">executeUpdate</span><span class="p">();</span><span class="w"></span>
<span class="n">ResultSet</span><span class="w"> </span><span class="n">rsKey</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">insertPs</span><span class="p">.</span><span class="na">getGeneratedKeys</span><span class="p">();</span><span class="w"></span>
<span class="n">Long</span><span class="w"> </span><span class="n">insertKey</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">null</span><span class="p">;</span><span class="w"></span>
<span class="k">if</span><span class="p">(</span><span class="n">rsKey</span><span class="p">.</span><span class="na">next</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">insertKey</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rsKey</span><span class="p">.</span><span class="na">getLong</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p><code>insertKey</code> is the primary key created for the corresponding insert, which can be used for the subsequent inserts.</p>
<p>As far as SQL is concerned.</p>
<div class="highlight"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">table1</span><span class="w"></span>
<span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">serial</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">c1</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">500</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">c2</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">500</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">c3</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">500</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">table1_pkey</span><span class="w"> </span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="w"></span>
<span class="p">);</span><span class="w"></span>
</code></pre></div>GIT post receive hook2012-07-20T10:00:00+05:302012-07-20T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-07-20:/blog/git-post-receive-hook.html<p>Whenever I make changes to my site, I end up doing lot of manual tasks deploying it to the production.</p>
<p>So I first moved the code to my self-hosted <a href="http://git-scm.com/">GIT</a> server. This reduced most of the manual tasks, however deploying were still tedious.</p>
<p>This is where GIT's <code>post-receive</code> hook came …</p><p>Whenever I make changes to my site, I end up doing lot of manual tasks deploying it to the production.</p>
<p>So I first moved the code to my self-hosted <a href="http://git-scm.com/">GIT</a> server. This reduced most of the manual tasks, however deploying were still tedious.</p>
<p>This is where GIT's <code>post-receive</code> hook came handy. <code>post-receive</code> is called when the repository receives changes. Adding a little piece of code solves the deployment issue.</p>
<p>In GIT hooks folder, create a file <code>post-receive</code> if not already available.</p>
<div class="highlight"><pre><span></span><code>$ <span class="nb">cd</span> path/to/git
$ <span class="nb">cd</span> repos/reponame.git/hooks
$ touch post-receive
$ chmod +x post-receive
</code></pre></div>
<p>add the following code to hook</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/sh</span>
<span class="nv">GIT_WORK_TREE</span><span class="o">=</span>/absolute/path/to/website git checkout -f
<span class="nv">GIT_WORK_TREE</span><span class="o">=</span>/absolute/path/to/website git reset --hard
</code></pre></div>
<p>In my case, I'm the only person who would be changing the repository and deploying it. So I do everything on master branch, and have not put much thought on having a development branch and production branch.</p>
<p>Be informed that, this hook is to just deploy and does not do any pre-deploy steps like preparing the environment or post-deploy steps like <code>collectstatic</code> or <code>dbsync</code>.</p>HTTP 406 response from spring controller2012-07-19T10:00:00+05:302012-07-19T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-07-19:/blog/http-406-response-from-spring-controller.html<p>On creating a <code>@ResponseBody</code> annotated spring controller that sends JSON, may lead to "406 Not Acceptable" response though the code/logic works fine.</p>
<p>The cause of this response is because, the controller cannot find the correct HTTPMessageConverter to satisfy the <code>@ResponseBody</code> annotated return value.</p>
<p>Check if, applicationContext.xml file has …</p><p>On creating a <code>@ResponseBody</code> annotated spring controller that sends JSON, may lead to "406 Not Acceptable" response though the code/logic works fine.</p>
<p>The cause of this response is because, the controller cannot find the correct HTTPMessageConverter to satisfy the <code>@ResponseBody</code> annotated return value.</p>
<p>Check if, applicationContext.xml file has</p>
<div class="highlight"><pre><span></span><code><span class="nt"><mvc:annotation-driven</span> <span class="nt">/></span>
</code></pre></div>
<p>and the following dependency is added to the pom.xml file.</p>
<div class="highlight"><pre><span></span><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.codehaus.jackson<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>jackson-core-asl<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.9.8<span class="nt"></version></span>
<span class="nt"></dependency></span>
<span class="nt"><dependency></span>
<span class="nt"><groupId></span>org.codehaus.jackson<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>jackson-mapper-asl<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.9.8<span class="nt"></version></span>
<span class="nt"></dependency></span>
</code></pre></div>RIP Stephen R Covey2012-07-17T10:00:00+05:302012-07-17T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-07-17:/blog/rip-stephen-r-covey.html<p>I am deeply saddened by the passing of Stephen R. Covey. His book "The 7 Habits of Highly Effective People" is one of the most influential and motivating book that has transformed my personal and work life into a breeze. I picked up this book when my work life was …</p><p>I am deeply saddened by the passing of Stephen R. Covey. His book "The 7 Habits of Highly Effective People" is one of the most influential and motivating book that has transformed my personal and work life into a breeze. I picked up this book when my work life was in chaos. Among few other books, articles and tools, this book had made my work life more effective and efficient. This book has taught me how to tackle a problem, how to prioritize and how to manage it.</p>
<p>I shall pray God to give his family and followers like me, all strength to overcome this great loss. May his soul rest in peace.</p>
<p><a href="http://bit.ly/O59Qcf">Press Release</a></p>How to set source level in Maven projects2012-07-16T10:00:00+05:302012-07-16T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-07-16:/blog/how-to-set-source-level-in-maven-projects.html<p>During developement... compiling or testing a maven project might lead to </p>
<div class="highlight"><pre><span></span><code>error: generics are not supported <span class="k">in</span> -source <span class="m">1</span>.3
</code></pre></div>
<p>Maven uses the 1.3 source setting as default, however it can be solved by explicitly mentioning the source level in project's pom.xml file.</p>
<div class="highlight"><pre><span></span><code><span class="nt"><build></span>
<span class="nt"><plugins></span>
<span class="nt"><plugin></span>
<span class="nt"><groupId></span>org …</code></pre></div><p>During developement... compiling or testing a maven project might lead to </p>
<div class="highlight"><pre><span></span><code>error: generics are not supported <span class="k">in</span> -source <span class="m">1</span>.3
</code></pre></div>
<p>Maven uses the 1.3 source setting as default, however it can be solved by explicitly mentioning the source level in project's pom.xml file.</p>
<div class="highlight"><pre><span></span><code><span class="nt"><build></span>
<span class="nt"><plugins></span>
<span class="nt"><plugin></span>
<span class="nt"><groupId></span>org.apache.maven.plugins<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>maven-compiler-plugin<span class="nt"></artifactId></span>
<span class="nt"><version></span>2.5.1<span class="nt"></version></span>
<span class="nt"><configuration></span>
<span class="nt"><source></span>1.7<span class="nt"></source></span>
<span class="nt"><target></span>1.7<span class="nt"></target></span>
<span class="nt"></configuration></span>
<span class="nt"></plugin></span>
<span class="nt"></plugins></span>
<span class="nt"></build></span>
</code></pre></div>
<p>However be reminded that setting the source level explicitly would solve the compile errors, but not having the appropriate runtime would lead to runtime errors.</p>Custom style and script as plugin2012-06-19T10:00:00+05:302012-06-19T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-06-19:/blog/custom-style-and-script-as-plugin.html<p>My first ever wordpress plugin <code>my-script-style</code>. After migrating my blog to wordpress, I wanted to include my own stylesheet and script files to do customization. There were many references online to put the files into the theme folder and edit the header.php to include them. However, I found that …</p><p>My first ever wordpress plugin <code>my-script-style</code>. After migrating my blog to wordpress, I wanted to include my own stylesheet and script files to do customization. There were many references online to put the files into the theme folder and edit the header.php to include them. However, I found that might not be a best practice as my files are independent of theme, and by doing so, I have to repeat this step every time I change my theme. I even found few on-the-shelf plugin's to do that.</p>
<p>Later, I decided to develop the plugin by my own, so that, I could get my hands wet on wordpress plugin's development. And I succeeded in this attempt. Source code could be found on <a href="https://github.com/arundhaj/wordpress">GitHub</a>. The advantage of this approach is, the plugin could be used irrespective of any theme.</p>
<p>This initial version just includes the css style sheets. I'm planning to enhance to support script files too.</p>Resize VirtualBox hard disk2012-06-18T10:00:00+05:302012-06-18T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-06-18:/blog/resize-virtualbox-hard-disk.html<p>Ever tried to increase the VirtualBox hard disk space?</p>
<p>I am running Ubuntu 12.04 as my guest OS on Windows 7 host with VirtualBox 4.1.16. To start with, I created a 8GB VDI as a dynamically allocated storage, so that I consume space only for what I …</p><p>Ever tried to increase the VirtualBox hard disk space?</p>
<p>I am running Ubuntu 12.04 as my guest OS on Windows 7 host with VirtualBox 4.1.16. To start with, I created a 8GB VDI as a dynamically allocated storage, so that I consume space only for what I use and grow as I need it. It was all fine until 7.5GB usage, and it was later I started receiving low disk space warnings. Later on exploring about the VDI, I found that, the dynamically allocated storage is supposed to grow until 8GB only. We have to increase the hard disk space. And to do that, (say, to increase hard disk space to 15GB)</p>
<div class="highlight"><pre><span></span><code>D:<span class="se">\V</span>irtualBox> VBoxManage.exe modifyhd Ubuntu_12_04_dev.vdi --resize <span class="m">15360</span>
</code></pre></div>
<p>Well, I did it and still receiving low disk space warnings. Hardly I see further steps mentioned after the above command. Fortunately, after long search I found a forum talking about it.</p>
<p><a href="https://forums.virtualbox.org/viewtopic.php?f=6&t=46814">https://forums.virtualbox.org/viewtopic.php?f=6&t=46814</a></p>
<p>It says, once the above command is issued, the increased hard disk space would be added as an unpartitioned disk. This has to be further partitioned and appended to the root partition. This could be done by any of your favorite partition tool. I've used Gparted to do so, as suggested in the same forum.</p>
<ol>
<li>Download the Gparted <a href="http://gparted.sourceforge.net/">http://gparted.sourceforge.net</a> ISO</li>
<li>Add the ISO as Virtual Disk. Boot from it using the same machine in Virtual Box</li>
<li>Delete the swap and then extend the first partition and make sure to leave enough at the end to add the swap back.</li>
</ol>
<p><img alt="Before partitioning" src="http://media.arundhaj.com/blog/resize-virtualbox/Gparted_before.png"></p>
<p><img alt="After partitioning" src="http://media.arundhaj.com/blog/resize-virtualbox/Gparted_after.png"></p>
<p>And here you go. All done.</p>Moving to GitHub2012-04-26T10:00:00+05:302012-04-26T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-04-26:/blog/moving-to-github.html<p>I've started hosting my personal source code repository onto GitHub, and could be found at <a href="https://github.com/arundhaj">https://github.com/arundhaj</a>. Those programs worth sharing to public would definitely find place in GitHub. However, I continue maintaining my private repository in BitBucket (<a href="https://bitbucket.org/arundhaj">https://bitbucket.org/arundhaj</a>).Happy coding and sharing.</p>Kernel source code mirror in Google2012-04-25T10:00:00+05:302012-04-25T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-04-25:/blog/kernel-source-code-mirror-in-google.html<p>Google is now hosting the public mirror of kernel source code (<a href="https://kernel.googlesource.com">https://kernel.googlesource.com</a>). This would facilitate more developers from reaching out to <a href="http://git.kernel.org">http://git.kernel.org</a>. Thanks Google for your support to open source community.</p>Productivity is about Product2012-04-17T10:00:00+05:302012-04-17T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2012-04-17:/blog/productivity-is-about-product.html<p>I have read an article which was contradicting the myth against fact.</p>
<p>It is generally believed that; Productivity of a developer is said to increase by reducing the development time using all possible tools and techniques. The clarifies that it is bound to fail at some situations.</p>
<ul>
<li>Sometimes spending less …</li></ul><p>I have read an article which was contradicting the myth against fact.</p>
<p>It is generally believed that; Productivity of a developer is said to increase by reducing the development time using all possible tools and techniques. The clarifies that it is bound to fail at some situations.</p>
<ul>
<li>Sometimes spending less time for a task is not the best thing for the task.</li>
<li>Just because you can use a control in the designer that does what you need (as well as everything else) doesn’t mean you should use it.</li>
<li>If taking up a new technology seems like a bad idea just because it has learning curve then maybe your priorities need readjustment.</li>
<li>Sustainable productivity is the only productivity that matters.</li>
</ul>
<p>The main goal as a developer is to create something great. This doesn’t always mean the easy or more familiar route. Sometimes it means the slow more complicated route or trying something new. Deciding the right tool for the job, really comes down to whether the product in productivity has something to gain.</p>
<p><a href="http://css.dzone.com/articles/developer-productivity">http://css.dzone.com/articles/developer-productivity</a></p>SMS game app on txtWeb2011-08-24T10:00:00+05:302011-08-24T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2011-08-24:/blog/sms-game-app-on-txtweb.html<p>I've created a SMS based game named FLAMES on txtWeb platform. txtWeb is an innovative product which brings the internet on SMS. For all the feature phones or the other kind of phones which doesn't have internet connection can still use this txtWeb to browse most of the commonly used …</p><p>I've created a SMS based game named FLAMES on txtWeb platform. txtWeb is an innovative product which brings the internet on SMS. For all the feature phones or the other kind of phones which doesn't have internet connection can still use this txtWeb to browse most of the commonly used sites.
Click here to know more about the <a href="http://www.txtweb.com">txtWeb</a>.</p>
<p>FLAMES is an age old game which is used to indentify the relationship between a boy and a girl. It is a simple app, which when sent an SMS with the names it would identify the kind of relationship that exists between these two. </p>
<p>Click here to know more about <a href="http://www.txtweb.com/apps/flames">FLAMES</a>.</p>
<p>The game is a web based app developed with Java Servlet and JSP and hosted in <a href="https://appengine.google.com">Google App Engine</a> (a cloud hosting of Google)</p>Consistency over heroism2011-06-14T10:00:00+05:302011-06-14T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2011-06-14:/blog/consistency-over-heroism.html<p>I’m sure everyone would remember their full-night studies in college for the exams next day. Whats more more surprising is to see the results come out with flying colors by clearing the papers and even getting good scores. This would make us feel heroic on achieving a target with …</p><p>I’m sure everyone would remember their full-night studies in college for the exams next day. Whats more more surprising is to see the results come out with flying colors by clearing the papers and even getting good scores. This would make us feel heroic on achieving a target with ease, while many others slog for the entire semester.</p>
<p>It makes us to wonder if the same technique could ever be possibly applied to our professional work too!!! Like, one full-night development a day before the deadline and bang, the application is released. A question that riddles me is that whether the application released like this is worth a release? Whats that make the exams and releases different? What is that evaluated in exams and that in applications?</p>
<p>Application does have different factors to be considered for evaluation. These factors are generally not unique for two different applications. This is because the desire differs from one customer to the other. What is considered obvious by one customer might not be even consider as a feature by the other. Of these many factors I see one factor common across all kinds of projects/service, yet very important one, the Communication Factor. Communication, I refer here is not a bullet-point in the feature list of the project; it is the one which directs the project.</p>
<p>Say a customer thinks a feature is as easier as graining a stone, however the actuals might be as hard as graining a mountain. If proper communication is established and by the deadline if the team has grained just 50% of the mountain would still keep the customer satisfied rather than with no proper communication and slogging many nights managing to grain 95% of the mountain would still dissatisfy the customer.</p>
<p>Being said that, would the heroic way of working would possibly ever satisfy a customer? Heroic way would give many last-minute surprises to the customers and it would definitly let them down. To satisfy, it is required to be consistent. Consistent in communication, consistent in delivery and consistent in everything for the entire project life cycle. This directly sets an example of a famous saying, "Results matters, efforts doesn't". We should learn to be smart enough to achieve results with less efforts.</p>
<p>P.S: All numbers mentioned here are to help understand and doesn't reflect actuals.</p>Boiler Plate2011-05-17T10:00:00+05:302011-05-17T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2011-05-17:/blog/boiler-plate.html<p>I used to wonder how the term boiler-plate penetrated into software industries. <a href="http://en.wikipedia.org/wiki/Boilerplate_(text)">Wikipedia</a> explains it as the part of the code duplicated multiple times within the project. Well, what I perceive is bit different; It is not called as boiler-plate because the code is duplicated, however it might be the …</p><p>I used to wonder how the term boiler-plate penetrated into software industries. <a href="http://en.wikipedia.org/wiki/Boilerplate_(text)">Wikipedia</a> explains it as the part of the code duplicated multiple times within the project. Well, what I perceive is bit different; It is not called as boiler-plate because the code is duplicated, however it might be the mental state of the programmer as if sitting over the boiler-plate fixing issues on this duplicated code.</p>License to use the word Productivity2011-02-23T10:00:00+05:302011-02-23T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2011-02-23:/blog/license-to-use-the-word-productivity.html<p>Lately I see a buzzword <em>Productivity</em> buzzing around everywhere within corporate, websites and blogs (even this blog is no exception). The word as such is not invented anywhere recently; it is in existence from 1612. I’m sure you are now wondering why this word got charm recently!!! All credit …</p><p>Lately I see a buzzword <em>Productivity</em> buzzing around everywhere within corporate, websites and blogs (even this blog is no exception). The word as such is not invented anywhere recently; it is in existence from 1612. I’m sure you are now wondering why this word got charm recently!!! All credit goes to Recession.</p>
<p>Well, very few people say they have to improve their personal productivity; in contrary many people say it is others who have to improve their productivity. Whoever be, the term Productivity applies to everyone in the corporate right from a team member till CXO’s of the company. This post is not about improving productivity but to explain what productivity is.</p>
<p>I got to quote a learning's from my friend: Earlier he used to stay late in office to meet the deadline which he is nearing to miss because of whatsoever reasons like wrong estimates or unforeseen risks and dependencies etc. After some how managing to achieve it, he proudly claimed to his manager that he has improved his productivity and reached the deadline successfully in spite of all hurdles which is otherwise impossible to achieve. He felt satisfied for his achievements.</p>
<p>It was then I explored the Personal Software Process (PSP), prioritizing tasks, assessing risks and dependencies, how relevant people have to be kept informed about the tasks and more importantly the value of each minute. These learning’s gave me the real insight of the term Productivity.</p>
<p>Mathematically, productivity is defined by the following formula;</p>
<blockquote>
<p>Productivity = Output quality and quantity / Input quality and quantity</p>
</blockquote>
<p>Productivity is said to be increasing only when the Output quality and quantity is increased with no or minimal change in the Input quality and quantity.</p>
<p>While I examine the earlier incidents with these learning, <em>it uncovered the fact that he was decreasing the quality to give a false impression of higher productivity.</em> The quality was degrading because of exhaustion of staying late nights, not prioritizing, not foreseeing risks and dependencies, unrealistic deadlines etc. Staying late nights is just to cover up someone’s mistakes, and not to increase productivity.</p>
<p>Hence the boundaries are defined for whom in a corporate are appropriate to use the term Productive. Be it a team member or a CXO’s, it is the one who only understand and increase the Output by maintain the steady Input are licensed to use <em>Productivity</em>.</p>Take Ownership or Give Ownership2011-01-18T10:00:00+05:302011-01-18T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2011-01-18:/blog/take-ownership-or-give-ownership.html<p>We should <em>Take Ownership</em>, that is how we could bring the best in the work we do. We take ownership of a work due to the passion over it. The passion gives energy and we take pride in getting it done. When we don’t have passion on the work …</p><p>We should <em>Take Ownership</em>, that is how we could bring the best in the work we do. We take ownership of a work due to the passion over it. The passion gives energy and we take pride in getting it done. When we don’t have passion on the work, we tend to deny ownership.</p>
<p>Taking ownership isn’t the same as taking responsibility. Taking responsibility is just the act of accepting a task and reporting back on completion. But taking ownership defines how we execute that task.</p>
<p>In my home, I take ownership in whatsoever work I do. Could it possibly happen in a corporate/team? In a corporate/team, one cannot take ownership, however it has to be given and it has to be defined. If it didn’t happen, it would lead to chaos.</p>
<p>Giving ownership is just how much we are open and transparent about the work we are assigning. If we don’t give ownership it is really hard to get it done. In a successful team, we could see the perfect blend of Giving ownership and Taking ownership.</p>
<p>Its all about Giving Ownership.</p>Real world object2011-01-12T10:00:00+05:302011-01-12T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2011-01-12:/blog/real-world-object.html<p>This is the first bullet line which every one would write when asked about Object Oriented Programming(OOPs). In college, when any student answers this, we could hear a huge laugh from every corner of the room teasing that the student has memorized as in textbook.</p>
<p>However the fact is …</p><p>This is the first bullet line which every one would write when asked about Object Oriented Programming(OOPs). In college, when any student answers this, we could hear a huge laugh from every corner of the room teasing that the student has memorized as in textbook.</p>
<p>However the fact is that, this is a simple and single line definition to OOPs. All other concepts and techniques just helps to match with real world objects. It is too hard to teach, or even talk about OOPs to the person who doesn’t realize this definition.</p>
<p>Well, consider a simple banking application. Except the building of the bank and papers; all others are intangible objects!!! Then how could they possibly be mapped with real world objects? So, it is important to understand that Real world object doesn’t mean some tangible objects. It is just how we assume and map it in our mind with some physical entity.</p>
<p>Once we are able to map it with some real world object, then we could easily write code to give life to that object; tell how to behave when alone and when teams with some other objects; how much information to share and how to share it; what to inherit from parents and what to give to children; and finally how to die.</p>Productive in home2011-01-11T10:00:00+05:302011-01-11T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2011-01-11:/blog/productive-in-home.html<p>All the time I used to wonder how people spend time working in home without distractions. The work I mentioned here is not the office work, however I was trying to study and do some hobby projects. Basically they are the personal work other than mails and social networking which …</p><p>All the time I used to wonder how people spend time working in home without distractions. The work I mentioned here is not the office work, however I was trying to study and do some hobby projects. Basically they are the personal work other than mails and social networking which I try to do with my laptop. I made a try by myself, but ended without any fruitful results. Then I made a research in internet to gather some tips from the people who work productively in home.</p>
<p>By reading those articles and tips, I realized that all the way I tried is actually the way one shouldn’t be doing. I used to work from my bed which is the first worst step to work at home. Like this, most of the steps I followed were not recommended by many people. With reference to all those articles, I compiled my list of things which I lack and the area I should put focus on improving them.</p>
<ul>
<li>Set up a better office space, a good table and chair would suffice. Never ever work on bed.</li>
<li>Make a plan before booting the laptop.</li>
<li>Don’t start with browsing emails and social networking sites.</li>
<li>Allocate time to chats, after that logout immediately. Don’t stay online forever.</li>
<li>It doesn’t mean that, we should completely keep away from family and friends. After all family and friends is our top priority in home. However keep them informed about your schedule.</li>
<li>Don’t work all the time in home. Spend time with family and friends.</li>
</ul>How much work is too much2010-11-09T10:00:00+05:302010-11-09T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2010-11-09:/blog/how-much-work-is-too-much.html<p>I had an interesting read in HBR’s Conversation with Bob Pozen. How he balances the work and leisure, the commitment to work and family.</p>
<p><a href="http://blogs.hbr.org/cs/2010/11/its_not_the_time_you_spend_but.html">http://blogs.hbr.org/cs/2010/11/its_not_the_time_you_spend_but.html</a></p>
<p>What we should get from this article is:</p>
<ul>
<li>Efforts aren’t matter, results count.</li>
<li>One can …</li></ul><p>I had an interesting read in HBR’s Conversation with Bob Pozen. How he balances the work and leisure, the commitment to work and family.</p>
<p><a href="http://blogs.hbr.org/cs/2010/11/its_not_the_time_you_spend_but.html">http://blogs.hbr.org/cs/2010/11/its_not_the_time_you_spend_but.html</a></p>
<p>What we should get from this article is:</p>
<ul>
<li>Efforts aren’t matter, results count.</li>
<li>One can become a top performer only when he realizes the importance of work and leisure.</li>
<li>Never procrastinate... Realize the importance of time and improve productivity</li>
</ul>Step 1 for compiling and burning2010-11-04T10:00:00+05:302010-11-04T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2010-11-04:/blog/step-1-for-compiling-and-burning.html<p>Everytime I search the net for the basic steps in compiling and burning my programs. So, I wanted to record it here.</p>
<p>I use ATmega32 microcontroller and USBasp programmer.</p>
<p>Compile</p>
<div class="highlight"><pre><span></span><code>$ avr-gcc -mmcu<span class="o">=</span>atmega32 -Os first.c
</code></pre></div>
<p>Convert to hex</p>
<div class="highlight"><pre><span></span><code>$ avr-objcopy -j .text -j .data -O ihex a.out first.hex …</code></pre></div><p>Everytime I search the net for the basic steps in compiling and burning my programs. So, I wanted to record it here.</p>
<p>I use ATmega32 microcontroller and USBasp programmer.</p>
<p>Compile</p>
<div class="highlight"><pre><span></span><code>$ avr-gcc -mmcu<span class="o">=</span>atmega32 -Os first.c
</code></pre></div>
<p>Convert to hex</p>
<div class="highlight"><pre><span></span><code>$ avr-objcopy -j .text -j .data -O ihex a.out first.hex
</code></pre></div>
<p>Before burning the hex file set the clock</p>
<div class="highlight"><pre><span></span><code>$ avrdude -p m32 -P usb -c usbasp -tuF
avrdude>sck <span class="m">10</span>
avrdude>quit
</code></pre></div>
<p>Burn it to the microcontroller</p>
<div class="highlight"><pre><span></span><code>$ avrdude -p m32 -P usb -c usbasp -U flash:w:first.hex
</code></pre></div>
<p>To erase the chip:</p>
<div class="highlight"><pre><span></span><code>$ avrdude -p m32 -P usb -c usbasp -tuF
avrdude>sck <span class="m">1000</span>
avrdude>e
avrdude>sck <span class="m">10</span>
avrdude>quit
</code></pre></div>
<p>PS: avrdude requires administrative privilege to execute, so it is suggested to use sudo command.</p>Foss tools I use2010-11-02T10:00:00+05:302010-11-02T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2010-11-02:/blog/foss-tools-i-use.html<p>For long time I used to talk about open source. A question stroke, about the open source softwares I use. I collected the list of softwares I use in Windows environment apart from the softwares I use in Ubuntu. Here it goes:</p>
<ul>
<li>Open office.org - an excellent office productivity tools …</li></ul><p>For long time I used to talk about open source. A question stroke, about the open source softwares I use. I collected the list of softwares I use in Windows environment apart from the softwares I use in Ubuntu. Here it goes:</p>
<ul>
<li>Open office.org - an excellent office productivity tools which has writer, calc, impress and more... and now I'm migrating to LibreOffice.</li>
<li>GIMP - a professional quality image manipulation tool.</li>
<li>WinMerge - file comparision and merge tool.</li>
<li>Firefox - browser, apart from this I widely use Google chrome.</li>
<li>Notepad++ - a rich functionality, light-weight editor.</li>
<li>StarUML - UML design tool</li>
<li>OpenProj - project management tool</li>
<li>Apache HTTP server and Tomcat - production webservers</li>
<li>MySQL and PostgreSQL - relational database</li>
<li>TortoiseSVN - versioning tool.</li>
<li>VLC media player - media player which runs almost all formats</li>
<li>CamStudio - for screen recording.</li>
</ul>
<p>Quite an impressive list. Still I'm widely looking at all possible options to use FOSS tools and improve this list. I'm proud to use FOSS tools and develop the FOSS community.</p>Atmel AVR Microcontroller2010-09-29T10:00:00+05:302010-09-29T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2010-09-29:/blog/atmel-avr-microcontroller.html<p>It is been two tiresome nights to sort out the issue of AVR USB Programmer. Really, Windows Vista is an OS which learners should never think of. Lots of efforts put on it went vain until Ubuntu came to my rescue.</p>
<p>The following GNU tools come in handy when it …</p><p>It is been two tiresome nights to sort out the issue of AVR USB Programmer. Really, Windows Vista is an OS which learners should never think of. Lots of efforts put on it went vain until Ubuntu came to my rescue.</p>
<p>The following GNU tools come in handy when it comes to AVR development in Ubuntu:</p>
<ul>
<li>gcc-avr : GNU C Compiler</li>
<li>binutils-avr : Binary utilities (linker, assembler)</li>
<li>avr-libc : C Library</li>
<li>gdb-avr : GNU debugger</li>
</ul>
<p>I'm using AVRDUDE to program my Atmel AVR MCU.</p>What am I doing here2007-02-22T10:00:00+05:302007-02-22T10:00:00+05:30ArunDhaJtag:www.arundhaj.com,2007-02-22:/blog/what-am-i-doing-here.html<p>Expressing self, that's why this site exist. I share things that I muse, I share since it makes me happy and I share so that I continue learning. I won't be focusing on any particular topic or category, however it is mix of all and still more.</p>