Ramblings on technology with a dash of social commentary
RSS icon Email icon Home icon
  • How to split a file into 2 chunks in bash

    Posted on May 9th, 2014 phpguru No comments

    I have a task to take a file with millions of lines, and to split it in exactly two files, with a controlled top `head` portion part A file, and the remainder of the lines in part B file.

    It actually turns out to be somewhat non-trival to split a file in 2 chunks, with a small defined top portion and a large arbitrary bottom portion.

    The split [OPTION]... [INPUT [PREFIX]] function is really designed to make N chunks out of your file, or M chunks each containing N lines; I debated using it and merging all but the first file back together, but decided to look up sed examples instead.

    I ended up with the below function. There’s almost certainly a faster/better way to achieve it but this seems to work.

    Sample usage

    remainderof bigfile.txt 1000
    When that runs you will have this result

    1000_bigfile.txt // this is the top 1000 lines
    1000R_bigfile.txt // this is the remainder of the file after splitting off the top 1000 lines
    bigfile.txt // the original untouched file

    Using optional 3rd argument

    remainderof bigfile.txt 1000 true
    When that runs you will have this result

    1000_bigfile.txt // this is the top 1000 lines
    bigfile.txt // this is the remainder of the file after splitting off the top 1000 lines
    // the original file no longer exists

    Here’s the function

    alias ll='ls -larth'
    
    function remainderof() {
    
        thefile=$1
        batchsize=$2
            if [ ! -z $3 ]
                    then
            : # $1 was given
            replaceoriginal=$3
            else
            : # $1 was not given
            replaceoriginal=false
            fi
    
        if [ $# -lt 2 ]
        then
             echo "Usage: $0 filename_to_split rows_to_chop [opt_bool_process_in_place_destructively]"
             exit 1
        fi
    
        extension="${thefile##*.}"
        filename="${thefile%.*}"
        length=$( wc -l < $thefile )
        buffer=$(($length-$batchsize))
        echo Splitting $batchsize lines off the top of $filename $extension leaving $buffer from $length  ...
        #  The top chunk filename
        topfile=${filename}_${batchsize}.${extension}
    
        # The bottom chunk = remainder of file
        startofnext=$(($batchsize+1))
        remainder=${filename}_${startofnext}-${length}.${extension}
        echo "Writing $topfile and $remainder"
    
        # Split off the first N lines of the file
        head -n $batchsize $thefile > $topfile
    
        # split off the bottom LENGTH - N lines of the file
        sed "1,${batchsize}d" $thefile > $remainder
    
        # whether to leave a copy
        if [ "$replaceoriginal" = true ] ; then
            rm -rf $thefile
            mv $remainder $thefile
        fi
    
        echo `wc -l < $topfile` lines in $topfile
        echo `wc -l < $remainder` lines in $remainder
    
    
        echo Done
    }
    

    If you have improvements, leave a comment!

  • New Kohana 3 Resources

    Posted on November 29th, 2010 phpguru No comments

    Kohana 3 is getting new fixes and upgrades every day. Here’s the latest batch of helpful new Kohana 3 resources on various topics.

    From Mixu.net…

    Kohana 3 Form Validation – validating form input (here’s another post on user input in Kohana 3)

    Kohana 3 i18n – localization/internationalization

    From Kerkness.ca…

    Kohana 3 ORM – Object Relational Mapping

    From DealTaker.com – now a 9 part Kohana 3 tutorial series

    Kohana 3 Routing – Choosing the right action based on the request

    From o1iver.net…

    Kohana 3 Introduction – 3 part series

    From elsewhere…

    Kohana 3 resized image urls

    Migrating to Kohana 3 from Dupal in 2 weeks

    Custom 404 page in Kohana 3

    Awesome explanation with diagram of (H)MVC

    Seems more are coming out every day. Kohana 3 rocks.

  • JQuery Timer Opens ColorBox…Then Flash Displays on Top in IE

    Posted on November 22nd, 2010 phpguru No comments

    I’m in the process of adding a new feature to the home page of a client’s website. The desired functionality is best described as follows:

    When visiting the home page, after a few moments, I want an overlay to appear, and inside the overlay will be a promotion to sign up for the website’s email newsletter.

    A friend of mine (o3c) recently turned me onto ColorBox, from Colorpowered.com. Excellent stuff! I’m almost done implementing it, and ran into a few integration issues that I thought I’d share. For the record, I want to state that none of the issues I ran into are problems with ColorBox! The ColorBox code is rock-solid. No, as usual, the problems are with Flash content on the page, and particularly within Microsoft Internet Explorer. As usual, everything works fine in Firefox, Safari & Chrome.

    So we have a couple of questions to answer. One is: How do you open ColorBox with a Timer? And the second is: How do you open a ColorBox over Flash in IE?

    Timers are still done with “old school javascript.” The syntax is a little goofy, but it’s not specifically JQuery related. We’ll come back to that.

    You have Flash on a page, and when your ColorBox appears on the page in IE, the Flash content underneath the overlay draws over top of the ColorBox overlay content, right? Contrary to how the rest of the world behaves (correctly), Adobe Flash (by default) draws last, which causes it to look like it’s on top of everything else. There are usually some easy fixes for that (most often, #3, below).

    Still reading? Ok, let’s get down to it. Here’s some general troubleshooting tips before we begin.

    1. Make sure your page Validates. Use the validator.w3.org
    2. Depending on your usage, you may have to make IE8 behave like IE7… a.k.a. force IE7 compatibility mode
    3. Tell the Flash content to have the “wmode” parameter, and set the value to “transparent

    Now, you may have already tried all of these things. I did, and the Flash content was still showing on top of my ColorBox content.

    So what’s a web developer to do. JQuery to the rescue. What we can do is hide the Flash content before the ColorBox appears, and turn it back on when the ColorBox goes away. It’s not a perfect solution, but it will hopefully work on all browsers so we can go about our day.

    In looking at the ColorBox API/Documentation, they’ve provided two events that look to be very helpful to this type of situation. Scroll down to the bit about “Extending,” almost to the very bottom where it says “Event Hooks” and look for the following:

    ‘cbox_open’ triggers when ColorBox is first opened, but after a few key variable assignments take place.
    ‘cbox_closed’ triggers as the close method ends.

    The container on my page that contains the Flash content has an id=’img_accordian’. My first thought was to do something like:

    $('#img_accordian').bind('cbox_open', function(){
       $(this).hide();
    }).bind('cbox_closed', function(){
       $(this).show();
    });

    but the above did not work! When I reloaded the page, it had no effect. No errors, but nothing happening. A quick Google search revealed this tidbit on the ColorBox Discussion Group on Google. So I just tried the more obvious…

    $().bind('cbox_open', function(){
       $('#img_accordian').hide();
    }).bind('cbox_closed', function(){
       $('#img_accordian').show();
    });

    Boo-yah! Now I can move onto something much more fun. So much of my time wasted due to inadequacies of Internet Explorer. Sigh… such is life.

    Whoops — not so fast — now that we have our signup form opening into a ColorBox iFrame, it shows every time we visit the home page. We want it only to show this pop-up-style signup form once during the user’s session. A cookie is the perfect place to store a simple “yes,” value, indicating that the user has seen the popup already.

    Another Google search on JQuery Cookie found the JQuery Cookie plugin. To make use of this, we need to open the signup form ColorBox only if the user has not seen it already. Then if we open it, we can set the cookie that says they saw it, thus preventing it from showing again (until we clear cookies or the cookie expires.)

    The two last features you’ll see in the code below are the finished timer, which I have set to open in 7.5 seconds. In testing, I may change it to 7, 6 or even 5 seconds. Also, I wanted a second “signup for e-news” link on the page in the footer that triggered the ColorBox overlay. So you’ll see in my code I created a custom object, “cbprops” to store the custom ColorBox configuration values, and then attached it to fire on the #signup_link.

    Here’s the final completed code.

    $(document).ready(function()
    {
        var cbprops = {opacity: 0.5, href:"/signup", width:"660px", height:"680px", iframe:true};
        if($.cookie('viewed_signup')!='yes')
        {
            var timer = setTimeout(function(){
            $.fn.colorbox(cbprops); }, 7500);
        }
        $("#signup_link").colorbox(cbprops);
    }).bind('cbox_open', function(){
        $('#img_accordian').hide();
        $.cookie('viewed_signup','yes')
    }).bind('cbox_closed', function(){
        $('#img_accordian').show();
    });

    There you have it, folks. We’re using a Javascript timer to open a ColorBox that hides Flash content in the background and setting a cookie so it only happens once. I hope you enjoyed reading — happy programming! Feel free to steal this code. If stealing doesn’t sit right with you, feel free to donate to JQuery or buy a 2011 Funny Pug Calendar!