Thursday, January 9, 2014

Converting Excel Date to mySql Date

I read about this trick from
http://blog.mclaughlinsoftware.com/2009/06/16/excel-date-conversion/

The trick is to convert the Excel date to the mySql format using the Text function.

=Text(A1,"yyyymmdd")

I want to keep the original date in the excel sheet, so I created a new column in the table and filled it with the above formula.

Next, I saved the spread sheet and then saved it again as a CSV file.

I removed the column names from the file, copied the column with the formula back to itself as values, and deleted the original date column.

I now have my original Excel file and a copy as a CSV that has the correct date format.

I then imported the CSV file into the mySql table.

Monday, December 30, 2013

Memory Leak Update

I have been using Hibernate with the default C3P0 connection pooling.

I have been receiving PERM GEN errors.

After reading Frank Kievet's blog about memory leaks, I started investigating.
http://frankkieviet.blogspot.com/2006/10/classloader-leaks-dreaded-permgen-space.html
http://frankkieviet.blogspot.com/2006/10/how-to-fix-dreaded-permgen-space.html

I added his code for causing a garbage collection in the perm gen and noticed that a lot of memory was being lost on each redeploy of the web app.

I downloaded the Eclipse, stand-alone Memory Analyzer (MAT).
http://www.eclipse.org/mat/

My steps to find the leak:
  1. I ran the web app and used MAT to obtain a heap dump from Tomcat. 
  2. I reran the web app again and obtained another heap dump. 
  3. I opened the dominator tree for each dump.
  4. The second dump contained the web app loader from the first dump. This class is causing the class loader memory leak.
  5. In the second dominator tree, I right-clicked the stale web app loader and selected Path to GC Roots (excluding weak references).
  6. The references listed are the ones causing the memory leak.
The leaks are usually for a third-party tool. It is important to close these tools properly. The tools I am using are Hibernate, C3P0 and MySQL. Each of these has to be closed properly. The best place for the code is in a SevletContextListener. I used several sites to find this information.
https://hibernate.atlassian.net/browse/HHH-7364
http://stackoverflow.com/questions/11872316/tomcat-guice-jdbc-memory-leak
http://docs.oracle.com/cd/E17952_01/connector-j-relnotes-en/news-5-1-23.html

I also had to update several jar files to newer versions:
hibernate-c3p0-4.1.1.Final.jar
mysql-connecto-java-5.1.28-bin.jar

package shared;

import com.mysql.jdbc.AbandonedConnectionCleanupThread;
import java.sql.Driver;
import java.sql.DriverManager;
import java.util.Enumeration;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import javax.servlet.annotation.WebListener;

@WebListener
public class WebappListener implements ServletContextListener {

    @Override
    public void contextInitialized(ServletContextEvent sce) {
    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        try {
            AbandonedConnectionCleanupThread.shutdown();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        try {
            Enumeration enumer = DriverManager.getDrivers();
            while (enumer.hasMoreElements()) {
                DriverManager.deregisterDriver(enumer.nextElement());
            }
        } catch (java.sql.SQLException se) {
            se.printStackTrace();
        }
        shared.HibernateHelper.closeFactory();
    }
}


The HibernateHelper class is a helper class for using Hibernate. All the methods are static. It has a static variable for the session factory that already exists.

    static public void closeSessionFactory(SessionFactory factory) {
        if (factory != null) {
            if (factory instanceof SessionFactoryImpl) {
                SessionFactoryImpl sf = (SessionFactoryImpl) factory;
                ConnectionProvider conn = sf.getConnectionProvider();
               
                if (conn instanceof C3P0ConnectionProvider) {
                    ((C3P0ConnectionProvider) conn).close();
                }
            }
            factory.close();
        }
    }

    static public void closeFactory() {
        closeSessionFactory(sessionFactory);
    }

The relevant imports for this method are:

import org.hibernate.SessionFactory;
import org.hibernate.internal.SessionFactoryImpl;
import org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider;
import org.hibernate.service.jdbc.connections.spi.ConnectionProvider;

Sunday, December 22, 2013

Starting with JSON

I am using jQuery in an application to build a user interface that creates a tree of classes. I want to use that tree as a template in another application that looks for a similar structure.

I am trying to use JSON to save the tree from the first app, read it in the second app, convert it to a Java object and then compare it to the other structure.

I took a quick look at JSON.simple, but I only looked at one post where the advice was to use either Jackson or GSON. I will try GSON first.

Download: http://code.google.com/p/google-gson/downloads/list

Tutorial: http://www.studytrails.com/java/json/java-google-json-parse-json-to-java.jsp

I have successfully completed my task.
  1. Created JSON from the object structure using jQuery and javascript.
  2. Added the JSON to a hidden field that is submitted to the server application.
  3. In the server application using Java and GSON, I read the JSON data and created a class hierarchy from it.
These are the methods that created the JSON from my object structure. There are two structures, TextPart and Group. Group contains TextPart and other Groups. TextPart contains text and another field to indicate if the text is equivalent to another TextPart.
    
    //grab the text, replacing " with \".
    function getTextPartText(textPart) {
        return $.trim($(textPart).children('code').text()).replace(/\"/g, "\\\"");     
    }
  
    //grab the text, replacing " with \". Change to lower case, change space to underscore.
    function getTextPartEquivalent(textPart) {
        var result = $.trim($(textPart).children('var').text().toLowerCase());
        result = result.toLowerCase().replace(/\s/g,'_');
        return result.replace(/\"/g, "\\\"");     
    }
  
    function textPartToJson(textPart) {
        return '{ ' +
                '"textPart" : "' + getTextPartText(textPart) + '"' +
                ', "equivalent" : "' + getTextPartEquivalent(textPart) + '"' +
               '}';
    }
  
    function groupToJson(group) {
        var state, result, children, comma, classAttr, i, radioGroup;
      
        state = "sequential";      
        radioGroup = $(group).children('.radioGroupType');
        if (radioGroup.length > 0) {
            if (radioGroup.children(".groupTypeRandom").prop('checked')) {
                state = "random";
            }
        }
        result = '{ "groupType":"' + state + '", "children": [';
        children = $(group).children();
        comma="";
        for (i = 0; i < children.length; i++) {
            classAttr = $(children[i]).attr('class');
            if ( classAttr === 'group') {
                result += comma + groupToJson(children[i]);         
                comma = ", ";
            } else if ( classAttr === 'textPart') {
                result += comma + textPartToJson(children[i]);                
                comma = ", ";
            } else {
                console.log("warn: skipping object in groupToJson: ", classAttr);
            }
        }
        result += "]}";
        return result;
    }

In the server application, create an equivalent object structure using GSON.

Object processJsonObject(JsonObject object) {
        if (object.has("children")) {
            Group group = new Group();
            group.setGroupType(GroupType.valueOf(object.get("groupType").getAsString().toUpperCase()));
            group.setChildren(processJsonArray(object.getAsJsonArray("children")));
            System.out.print("---start group");
            System.out.println(String.format(" (%s) ---", object.get("groupType").getAsString()));     
            processJsonArray(object.getAsJsonArray("children"));
            System.out.println("---end group---");
            return group;
        } else if (object.has("textPart")) { 
            TextPart textPart = new TextPart();
            textPart.setTextPart(object.get("textPart").getAsString());
            textPart.setEquivalent(object.get("equivalent").getAsString());
            System.out.print(object.get("textPart").getAsString());       
            System.out.println(String.format(", %s", object.get("equivalent").getAsString()));
            return textPart;
        } else {
            return null;
        }
    }
   
    Object[] processJsonArray(JsonArray array) {
        ArrayList list = new ArrayList();
        for (JsonElement element : array) {
            if (element.isJsonArray()) {
                list.add(processJsonArray((JsonArray)element));
            } else if (element.isJsonObject()) {
                list.add(processJsonObject((JsonObject)element));
            } else {
                System.out.println(String.format("No object or array: %s", element));
            }
        }
        return list.toArray();
    }

I added a serialize and deserialize routine for saving the object structure. This allows the structure to be referenced by other applications.

public void serialize() {
      try
      {
         FileOutputStream fileOut =
           new FileOutputStream(context.getRealPath("/WEB-INF/templates/example.rote"));
         ObjectOutputStream out = new ObjectOutputStream(fileOut);
         out.writeObject(getRoot());
         out.close();
         fileOut.close();
         System.out.printf("Serialized data is saved in /WEB-INF/templates/example.rote");
      }catch(IOException i)
      {
         i.printStackTrace();
      }
    }
   
    public void deserialize() {
        try
      {
         FileInputStream fileIn =
           new FileInputStream(context.getRealPath("/WEB-INF/templates/example.rote"));
         ObjectInputStream in = new ObjectInputStream(fileIn);
         setRoot((Group) in.readObject());
         in.close();
         fileIn.close();
      }catch(IOException i)
      {
         i.printStackTrace();
         return;
      }catch(ClassNotFoundException c)
      {
         System.out.println("Example class not found");
         c.printStackTrace();
         return;
      }
    }

In the other application, I read the file that was created by the first application.








Friday, December 20, 2013

Using jQuery

Download jQuery form http://jquery.com/download/

Add a script tag to the HTML page for jQuery. I am using Tomcat, so I am using JSP.

    <script src="${pageContext.request.contextPath}/jquery/jquery-2.0.3.min.js">

Place some jQuery inside another script tag.

        <script>
        $( document ).ready(function() {
            console.log( "ready!" );
        });
        </script>

Found a great tool for playing with jQuery: http://jsfiddle.net/

Thursday, December 19, 2013

Kiteboard Helmet Search

I am looking for a helmet for kite boarding. I have a large head. With short hair, my head is 24-1/4". My local stores do not have helmets that fit. I am looking for a helmet that fits and has good safety features.

I read this from an excellent post:

"Liner: There are three main types of insulation. Expanded Poly-Propylene (EPP) is the most common. It’s very stiff, not affected by moisture, and will take multiple impacts, according to Halstead. Expanded Poly-Styrene is "the very best" energy attenuating substance. It’s what’s used in bike helmets, but generally is only expected to be hit once, and with one exception noted below isn’t used in whitewater helmets. Finally, there are closed-cell foams. The best of these are otherwise known as Vinyl Nitrile, which is a PVC synthetic rubber, heavier in a safe density than EPS/EPP, and will absorb water, but it is usually much softer. A closed-cell foam will also absorb sweat along with water, which can make for a ‘funky’ brain bucket. A fourth insulation, Ethyl Vinyl Acetate or EVA, "isn’t a functional energy attenuator because it’s so springy," says Halstead.

Halstead says stiffer is better. "When you look at the stuff that will actually protect your head, it ought to be so stiff that you think, ‘my God, this is going to hurt.’ But you need that much stiffness to spread the impact. If you pick up a helmet with soft foam inside, the chance of it protecting you is slim." Of course, not everyone agrees with Halstead. Jay Norfleet from Grateful Heads thinks EPP is not a multiple-impact foam because it compresses. And Doug Poe of Pro-Tec believes EVA gives the same protection as closed-cell foam."

It is impossible to find a water helmet that fits a large head. My head is 62.5 cm. That is the upper limit of some helmets and above the limit for most.

Liquid Force XL 24in

ShredReady makes some helmets that are one-size-fits-all. I bought the Standard cut helmet and removed all the padding. I also cut away some of the mesh liner. It fits, barely.

Saturday, September 7, 2013

Using Yii Framework

I have downloaded the Yii plugin for NetBeans, but first I had to install NetBeans 7.3.

Download the .nbm file for the plugin.
Go to Tools -> Plugins and add a plugin.
Navigate to the location of the .nbm file.

Immediately after installing the plugin, I noticed that there were some updates for it.
After update, Yii was no longer installed. Actually, it is still there, but it is part of PHP. Click 'show details' to see that Yii is installed.

Yii documentation is asking me to set the path to yiic.php. I cannot find it on my Mac. Ignored it and continued.

Create new PHP project in NetBeans and set the framework to Yii. It is taking a loooonnnngggg time to create the project. After 5 minutes, I closed the new project wizard. Lather, rinse, repeat. I am able to open the project. When I view properties, it indicates that yii is part of the project.

I am trying this tutorial: http://www.larryullman.com/series/learning-the-yii-framework/

Now I see. The plugin is not Yii. Yii must be installed separately. Download Yii from http://www.yiiframework.com/download/. Lather, rinse, repeat.

In NetBeans, set the path to yiic.php from Preferences -> PHP -> Yii.

Create new PHP project, select PHP framework. Project has been created with lots of files.

Uploaded framework and requirements folder to remote PHP server. I accessed /yii/requirements to test the PHP installation for conformance with Yii. All passed, with some warnings for databases that I am not using.

PHP does not run out of the box with NetBeans. It is necessary to install PHP, Apache and MySql. Download MAMP at http://www.mamp.info/en/index.html.

Tutorial for installing: https://netbeans.org/kb/docs/php/configure-php-environment-mac-os.html.
There is no MYSQL option in the Services -> Databases. Trying to add new connection.

I already had a MYSQL server set up. I deleted it and was able to access the server config window.

Set up admin properties for MAMP, not MySql tool. Am able to start/stop mysql from IDE.

Created PHP project. Created index.php in Source Files. Ran project. Index file appeared.

Created a Yii project. Be sure to set the path to yii.php or the require statement will not work and the page will be empty.

Configured Yii to use Gii to create models, views, controllers. When creating the model, I had to rename some columns in the database that had embedded spaces.

http://localhost/appname/index.php/gii/default/index

Gii created the model, but only in the destination folder, not in the NetBeans source folder.

Created new NetBeans project in htdocs of Apache. Now I can use Gii to generate code. I do not like using htdocs. I will add alias that points to location of PHP projects.

Alias /php "/Users/xxx/yyy/zzz/php"

I had to stop Apache before adding the alias.

I edited the URL for the configuration (config/main.php) for the PHP project to include the new alias.

When I use Gii to generate code from the URL, the NetBeans project is updated. The source for the project is in the user folder, not in the MAMP application folder.

I bought the book Yii Application Development Cookbook Second Edition by Alexander Makarov. It has more information than the above tutorial. The tutorial allowed my to get my app running, but the book is showing me the inner workings of Yii.

Be careful not to set up a general route like => website/page, as it will prevent gii from running.

 MAMP has phpMyAdmin.

Edit the protected/components/UserIdentity.php file to add authenticated users. Modify the accessRules in each controller to give access rights to each user.

Modify .htaccess

Modify .htaccess in the root of the project folder, so that index.php does not have to be added to each URL. I had to modify the Apache server httpd.conf to allow options to be overridden in .htaccess.

Options +FollowSymLinks
RewriteEngine on

RewriteBase "/alias-to-php-projects/"
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule (.*) php-project-name/index.php/$1

Place this in the .htaccess folder that hosts the yii application.

Alternate Rewrite

Instead of repeating this in all the .htaccess files, I used a Directory tag in httpd.conf for Apache. I created an Alias to a folder that contains all my Yii projects, then add the rewrite for that folder.

  Alias /php "/Users/xxx/yyy/zzz/php" 

   
   
        Options +FollowSymLinks
        RewriteEngine on

        RewriteBase "/php/"
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_FILENAME} !-d  

        RewriteRule ^([^/]*)/(.*)$ $1/index.php/$2


   


The regex strips out the first part of the path and treats it as the name of the project, then appends index.php, then appends the rest of the path.

 Revisited Several Months Later

Several months after the initial controller, I wanted to create another one. Of course, I had forgotten everything!

Important files:
  • components/UserIdentity for adding users
  • config/main.php for defining URLs and other configurations
  • controllers
  • models
  • views

Renaming PHP project for NetBeans

To rename the PHP project, I had to copy the project to a new folder with the new name, then use the new project in NetBeans and delete the old one.

Change the project URL in the Run Configuration of Properties.

This does not work for a yii project, since the model was created for the original app.


































Thursday, August 22, 2013

MIME_PDF missing from org.apache.fop.apps.MimeConstants

I am building a Java app that uses Apache FOP.

I downloaded the source from http://xmlgraphics.apache.org/fop/download.html

I tried creating an instance with
            Fop fop;
            fop = fopFactory.newFop(MimeConstants.MIME_PDF, out);

I received the error that MIME_PDF did not exist.

I neglected to include the xmlgraphics-commons jar that is in the build folder of the fop install.

These are the jar files I needed to include:

xmlgraphics-commons
avalon-framework
fop

Next, I received an error for any XSL files that used the ends-with xpath function. I needed two more jars from the fop build folder.

serializer
xalan

More Blogs

Followers