Tuesday 5 November 2013

RevoDrive 3 X2


Just bought the RevoDrive 3 X2 480GB for my photo/video editing system, even though I read that random data performance is not great.  

I've had 3 other HDs in the computer for a while now: 

1. Corsair Force 3 SSD - 120 GB
2. Western Digital Caviar Green (WD30EZRX) - 3 TB 
3. Adata SSD S510 - 120GB

My setup has the Corsair as my system drive; Adata as the cache/scratch/paging/preview etc for Photoshop, Lightroom, Premiere, Resolve,Windows; WD Caviar as the storage drive.

My data flow is to import media on to the WD Caviar 3TB; if I decide to do a video project I copy the media to the Adata.  Since starting to work more with video, the 120 GB just isn't enough...it is for working on one video project at a time but that's no fun!

Initially I thought I'd buy two/three SSDs and stripe them together.  As they would not be the main storage, I could lose the drives and at worst I would lose time, but the original media would be fine (yes, I backup my 3TB drive religiously).  I knew of PCIe SSDs of course, but always assumed that they'd be way over budget.  I then came across a refurbished 480GB OCZ RevoDrive 3 X2.  Cheap!,  less than buying 3 SSDs and better performance (in most areas).

Here are the performance stats for the original drives (the Adata wins, though that could be down to fact that I'm using the Corsair as a system drive...the other difference is that I have the corsair partitioned, for various reasons):

Corsair:


WD Caviar (was seriously thinking about stopping the test, it was so slow!):



Adata:


And now the RevoDrive:

...shines bright like a diamond (sorry, been listening to Rihanna recently!)

Under random data tests:

Adata:


Revodrive:


Gotta admit I was expecting much worse after reading several reviews!  I'm certainly not getting the advertised 1,500 MB/s (never expected to), nor am I getting what others seem to be getting...but so chuffed with it I'm grinning like a Cheshire cat.  

I'll be doing some real world tests over the next few days, as that is where it really counts of course, along with testing various setups to see which works best.  Initial I'm gonna go with video related stuff (Resolve and Premiere) on the RevoDrive, with the Adata used for Lightroom, Photoshop, and Windows paging file.

Overall, really happy with the refurbished RevoDrive; packaging was immaculate, everything was sealed with no sign of it ever being used previously; install was smooth, just install the drivers and windows recognises it fine, format and away you go.


Monday 14 October 2013

CPU Overclocking



In my first weekend of CPU tweaking, I've managed to overclock my Intel i7-3820  from the stock 3.60GHz to 4.6GHz.  The temp under load has so far not gone above 56degC, so I've plenty of overhead to play with I think.  Very happy with that as I'm just using fan cooling.  Also, under heavy colour grading tests, the system is solid, no issues whatsoever (though the SSDs are still a painful bottle neck...really need to raid something together or get PCIe SSD...that would be blazingly fast).

Will do a detailed system set-up once I have managed to tweak a bit more and am happy to run with it full-time.  Gonna do a 6 hour CPU burn test this weekend, to be absolutely certain this is rock solid.

Friday 27 September 2013

Colour Grading and Sharpening

I loved working in a darkrooom, and is the thing I miss most about photography...nothing nowadays comes close, except maybe post production where you can tweak to your hearts content (or fix badly taken photos, as is too often the case!).

Recently, I've been tackling two basic post production topics: Colour Grading videos and Sharpening video and photos.

Colour Grading
Wow, what an interesting field.  The following quick video shows how much work it takes to make something simple look great, let alone the work that would be required for something like the hobbit!:




The "Camera original" shot (bit flat looking, but that is on purpose to enable better colour grading and maximise information capture i.e. dynamic range) was pretty darn good, until compared to the final.  What a difference!

I've been practising colour grading a little, but only in Photoshop and Lightroom as I don't have something like Premier.  Recently I've started looking at DaVinci Resolve.  Amazing for a free product, but is hellishly complex...can't wait to get my hands really dirty on it.

Sharpening
Different schools of thought, no definitive answer on whether its better to allow the camera to sharpen or sharpen in post.

Camera sharpening - the general theory is that sharpening in-camera is done before compression so that the sharpening is only done on the actual details rather than compression artefacts.

Post Sharpening - argument for this is that the sharpening algorithms used in camera is geared towards speed and not quality and you have more flexibility in post.  I don't know whether the first part about speed is true, but I certainly prefer the flexibility of post sharpening.

My own testing and prefered workflow has meant that I prefer to shoot with sharpening in-camera switched off and do any sharpening in post.  I like having full control over sharpening.  Same reason I shoot in Raw, I like having as much control over temp., black levels, white levels, etc etc.  I'm not a sports photographer, or shooting for clients who want to see results quickly, I just shoot because I like to.  I can afford to spend time in post and enjoy post production.

It's true that in most cases I use presets in post, which brings the sharpening to around what the camera would have done anyway, but it gives me scope to tweak the photos I really care about.  If it was set in camera I wouldn't get that option.

Tuesday 24 September 2013

Windows 8 Rant


So far the only good thing about Windows 8 is that I'm not using it on any of my desktops!

I thought it would be a good idea to upgrade my old laptop (Dell XPS M1330) before I handed it over to my daughter (just turned 7).  The laptop was running Vista Ultimate (:-0), so didn't think it was fair for her to deal with that.

So off I went; downloaded the Microsoft upgrade assistant and let it rip.  It came back to say, "yeah you're good to go but will have to reinstall a bunch of programs".  No problems for me as I wanted a fresh install anyway.  So went through the buying process; smooth.

The install process was just as easy, the only thing I think it asked me was whether I wanted to keep my settings to which I said no as I wanted a clean system.  The installer downloaded a 2 GB file, booted a couple of times and it was done (after I created my account that is).

The first frustration came when I tried to create an account for my daughter.  I wanted a Microsoft tied in account so that her settings would be saved in the "cloud" (what a dreadful term).  I also wanted protection and overview of her account.  So when creating her account, I told Microsoft her correct age.  When I logged in with her account on the laptop, Microsoft picked up on that and said, "you're a minor, please get parent to sign in"...all good.  So I signed in with my account and it said to verify that you are over 18 we'll carry out a $0.50 credit card transaction; no issues.  However when it redirected me to the payment site, the site was down.  After 30 mins, the site was still down.  This was becoming frustrating now as I couldn't get her email (google) working until I tied in her account.  it then started to work, so I got my CC out...another problem....I could not change country, so the address could only be american.  The frustration got to me at this point so I created a normal account and monitor on the local computer.

Yesterday I hit another frustration, and this is the one that has really got me wound up.  The Upgrade Assistant has installed a 32-bit OS not 64-bit!  Windows recognises the system as 64-bit capable, and I told it I didn't want any settings kept, but it still installed a 32-bit OS without giving me any indication it was doing that....what the hell!  If I had known it couldn't install 64-bit I would have stpped and done it differently.  To add to the frustration, it doesn't seem very easy to install the 64-bit version: http://winsupersite.com/article/windows8/windows-8-upgrade-32bit-64bit-144649 .  Oh and this morning, my daughter pointed out that the system just shuts down completely every so often...I witnessed this myself...never happened on Vista previously.

I don't want to go into the usability aspects of windows 8 (an OS made for touch screens that is really, really non-intuitive) as that is another level of frustrations, as that would just make me cry.


Thursday 5 September 2013

Amahi 7 and Greyhole on HP Proliant (N54L) Microserver

I've had a long winded ride in getting Amahi 7 with Greyhole installed and working.

Some of the problems I faced:
  • Amahi 7 supports only installing from a DVD; I don't want a dvd drive
  • Amahi 7 USB install guide did not work for me
  • Amahi 7 does not have greyhole integrated yet, so a manual install has to be done
  • hda-greyhole did not work well with Fedora 19's Samba
  • After installing hda-greyhole and configuring samba to use it, I could no longer access the share and was getting the following errors:
Sep  5 01:48:23 amahi smbd[13392]:   Error loading module '/usr/lib64/samba/vfs/greyhole.so': /usr/lib64/samba/libsmbd_base.so: version `SAMBA_4.0.0' not found (required by /usr/lib64/samba/vfs/greyhole.so)
Sep  5 01:48:23 amahi smbd[13392]: [2013/09/05 01:48:23.379707,  0] ../source3/smbd/vfs.c:184(vfs_init_custom)
Sep  5 01:48:23 amahi smbd[13392]:   error probing vfs module 'greyhole': NT_STATUS_UNSUCCESSFUL
Sep  5 01:48:23 amahi smbd[13392]: [2013/09/05 01:48:23.379830,  0] ../source3/smbd/vfs.c:349(smbd_vfs_init)
Sep  5 01:48:23 amahi smbd[13392]:   smbd_vfs_init: vfs_init_custom failed for greyhole
Sep  5 01:48:23 amahi smbd[13392]: [2013/09/05 01:48:23.379946,  0] ../source3/smbd/service.c:632(make_connection_snum)
Sep  5 01:48:23 amahi smbd[13392]:   vfs_init failed for service Music
Sep  5 01:48:23 amahi smbd[13392]: [2013/09/05 01:48:23.389801,  0] ../lib/util/modules.c:56(load_module)

Here is a summarised list of how I got things working.  I've had to recreate all the steps here, leaving out all the incorrect attempts and dead ends and trying to patch them in an order that might work, so this might (in fact likely) to  have errors, so be careful, here be dragons.

Standard disclaimer applies; use the details here at your own risk, you could certainly destroy all your data and risk a bricked system! Also, this is outside of Amahi supported practices.


Note
I run a headless system so all commands after the installation of Fedora, are run in a bash terminal.

I ran all commands as root (i.e. I executed sudo su – in the terminal first)

This assumes a fair amount of knowledge, like being able to use ping and edit files.

In a nutshell
  1. Upgrade Memory
  2. Update Bios
  3. Install Fedora 19
  4. Install Amahi
  5. Add Hard Drive
  6. Change Share locations
  7. Install Greyhole
Upgrade Memory
I installed more RAM (Corsair CMV8GX3M1A1333C9 Value Select 8GB (1x8GB) DDR3 1333 Mhz CL9 Mainstream Desktop Memory Module”) first after taking out the 2GB it came with as I prefer my RAM to be exactly the same in my servers.

The case is easy to take apart; after you take off all the cables connected to the motherboard (conveniently easy to get to as it is at the front) the motherboard comes out easily as it sits on a metal sliding base.  The only issue I had was taking off the two thumb screws holding the motherboard in place as they were too tight to remove by hand…but HP provide a handy L-shape star wrench to use.

Update Bios
The official bios for this board has been modded to support 6 ports of AHCI SATA 3gb/s.  You can find the modded bios here: http://www.avforums.com/forums/networking-nas/1521657-hp-n36l-n40l-n54l-microserver-updated-ahci-bios-support.html

To create a bootable USB key with the bios just download the .rar file, extract, and execute RunMe.exe.

Install Fedora 19
Prepare your USB stick

Download Unetbootin: http://unetbootin.sourceforge.net/unetbootin-windows-latest.exe
Unetbootin is a standalone executable so doesn't need to be installed.  You will need admin rights.
Make sure you have a USB stick with enough space in your machine, then execute UNetbootin.
In the UNetbootin window for Distribution select Fedora 19_Live_x64

Make sure under type at the bottom, "USB Drive" is selected and that next to Drive it shows your USB stick, else you run the risk of overwriting your data.
Once UNetbootin finishes, don’t reboot, just exit.

Remove your USB stick and boot off it on your server.  By default it should be able to boot off USB, however you may need to enable booting off USB in the Bios if that is not already set .

Complete Fedora installation by following the instructions that appear.

Log on, turn on SSH and make available after reboot:
service sshd start
chkconfig sshd on

DNS
Make sure you can connect to the internet (ping google.com for example).  If you can’t, try manually adding DNS servers to /etc/resolv.conf (you could add the openDNS servers for example: 208.67.222.222 and 208.67.220.220)

Install Amahi
To install Amahi execute:
rpm -Uvh http://f19.amahi.org/noarch/hda-release-6.9.0-1.noarch.rpm
yum -y install mariadb-libs mariadb-server
yum -y install hda-ctl hda-platform
hda-install 

If all went well, your Amahi control panel (https://www.amahi.org/users) should show your new install as running (might take a minute or two to catchcup)
Optionally you can turn off DHCP in your router at this stage

Add Hard Drive
I used parted for prepping my new 4TB hard drive, which appeared as /dev/sdb (read here: http://rainbow.chard.org/2013/01/30/how-to-align-partitions-for-best-performance-using-parted/)
parted /dev/sdb
at the parted prompt
mklabel gpt
mkpart primary ext4 4096s 100%
align-check opt 1
quit

Format
mkfs.ext4 -L greyhole01 /dev/sdb1
mount
hda-diskmount
using the output from the above command, first backup /etc/fstab, and then add to /etc/fstab.
For me this meant I added the following to the bottom of my fstab:
UUID=e68bc54a-b795-4ce0-802f-213f4ab71e60 /var/hda/files/drives/drive1 ext4 defaults 1 2

Change Share Locations
I changed my default share paths from the drive that the OS lived on to my new drive using the Amahi web interface (after creating the following folder /var/hda/files/drives/drive1/shares/).  This is so I have plenty of space before greyhole can move files and as it is a drive in the pool, when Greyhole moves the files it should happen quicker if it decides to store a copy on the same drive (of course copies that need to be moved to other pooled drives will take more time).

Install Greyhole
As I’d already followed the installation instructions for installing greyhole on Amahi, before finding out that Amahi 7 didn’t work, I didn’t have to recreate the DB schema.  Here is what I had previously done:
yum -y install hda-greyhole
hda-create-db-and-user greyhole
mysql -u greyhole -pgreyhole  greyhole < /usr/share/greyhole/schema-mysql.sql

To upgrade greyhole, I removed hda-greyhole first:
yum remove hda-greyhole

and installed it from the greyhole repository
curl -o /etc/yum.repos.d/greyhole.repo http://www.greyhole.net/releases/rpm/greyhole.repo
yum update greyhole

I edited the /etc/greyhole.conf file, adding in my storage_pool_drive:
storage_pool_drive = /var/hda/files/drives/drive1/gh, min_free: 10gb

...and changed the db_user and db_pass to (security minded people should change this at some point):
db_user = greyhole
db_pass = greyhole


In /etc/samba/smb.conf I added the following two lines to the shares I wanted to participate in the pool:
dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

I then restarted Samba and greyhole
Service smb restart
Service greyhole restart

I now have a working Amahi 7 with greyhole system.  I’ve tested it and all seems to work very well.


In summary
The combination that did not work:
Samba: samba-4.0.9-1.fc19.x86_64
Amahi Greyhole: hda-greyhole-0.9.31-1.x86_64

The combination that does work:
samba-4.0.9-1.fc19.x86_64
greyhole-0.9.34-1.x86_64

I'm fairly sure I've made a few mistakes in creating this both in terms of steps taken to get Amahi+Greyhole running, and producing this document.  Please feel free to, constructively correct me :-)

Links
Some links that gave me solutions and others that helped get me pointed in the right direction

Amahi Install related
https://wiki.amahi.org/index.php/Amahi_7_installation
https://wiki.amahi.org/index.php/Express_CD
https://wiki.amahi.org/index.php/USB_Install
http://docs.amahi.org/amahi-7-express.html

Greyhole related
https://wiki.amahi.org/index.php/Greyhole
http://greyhole.googlecode.com/svn-history/r415/trunk/INSTALL
https://forums.amahi.org/viewtopic.php?f=39&t=2714&p=13570
https://getsatisfaction.com/greyhole/topics/after_upgrade_now_shares_seem_to_have_stopped_working
https://forums.amahi.org/viewtopic.php?f=39&t=4035

HD Installation related
https://wiki.amahi.org/index.php/Adding_a_second_hard_drive_to_your_HDA
http://rainbow.chard.org/2013/01/30/how-to-align-partitions-for-best-performance-using-parted/

General reading
http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/VFS.html

Too many others to list

Tuesday 3 September 2013

HP Microserver N54L and Amahi

59-107-052-Z01
I bought a HP Microserver a little while ago to replace my 15 year oldish raid server.  I will be going from a full tower (around 60cm tall), 2.7 TB, 8 drive raid 5 array with 3ware raid controller (with OS on separate HD of course) running  Gentoo to a HP Microserver coming in at just 27 cms high, 4 TB on a single drive (for starters), with capacity for 3 other HDs (could have more if I use the 5.25 slot for more HDs...how things change.  

Yesterday night I upgraded the paltry 2GB RAM it came with to 8GB RAM (system can handle 16 GB RAM apparently, might bump it up in future).  The server is super easy to take apart and get to the motherboard. The HD bay is also really convenient to get to, so when the 4TB HD comes through, it should be a breeze. 

I went for the Hitachi 4TB Deskstar (7K4000) in the end.  It was the top 4TB recommendation in Toms Hardware tests back in April 2012 and is at a price point within my budget now for the performance. This arrives today to be installed tonight after the bios firmware upgrade and Amahi install.

There are actually a few modded bios firmwares available for the HP (see below link).  The one I'm interested in is the one that allows the 6 USB ports to have AHCI enabled.  As it will be a server, I like to update the bios to the latest before I install any software, OS or otherwise, so this will be the first thing I do.  Luckily both the official and modded firmware are built to be done from a USB stick, so don't foresee any extra work there.

Once the firmware is updated I'm gonna be installing Amahi for all the NAS type things like sharing files, media, music and a whole host of other things.  

Amahi recently changed their install process from "install OS yourself, then install Amahi" to "here is an ISO that you need to burn to DVD and do the installation".  Currently that is the only supported method.  I don't have a DVD reader (and don't intend to have one as I have very little use for one) so can't use that method.   However, they have published a how-to for all us unsupported types, so hopefully will have the system up and running tonight.  Then I can copy across the existing 1.8 TB worth of data, backup to Bitcasa, and decomm. my 15 year old server...tears will be shed.


Links




Friday 30 August 2013

TV Calibration Challenges

It would have been easier if I had an expensive spectrum analyser or independent TV calibration hardware to make sure the TV is set up correctly in the first instance but oh well...

Setup
I have the following sources, all via HDMI directly into the Panasonic P65VT65:

1. SkyHD
2. XBMC on Acer Revo on XBMCBuntu
3. CCTV
4. Blu-Ray player (in future this will share the HDMI TV input with the CCTV via a HDMI switcher as the TV has only 3 HDMI inputs)

CCTV, I'm not going to worry about at all.  
Blu-Ray player I'll do when I get it (though it could help me calibrate the TV as it could act as the reference source).

SkyHD
The SkyHD box is a problem as I can't load test images on it.  Sky used to send out a test card (replicating the old BBC test card but updated with Myleene Klass):



They do not send it out any longer it seems.

BBC themselves used to broadcast a specific HD test card:


They too have stopped.  

So the only thing I can do is calibrate Sky by eye, not the most perfect or elegant solution.  I've set this to THX Cinema mode (very happy with this mode in general) and tweaked the contrast and brightness a little.

XBMC on Acer
There are a few things that complicate this.  As I'm running XBMC on a Linux PC (XBMCbuntu) I have to take in to consideration the graphics card (Nvidia) and drivers and software on top i.e. Linux and XBMC.  Each of these could have an effect.  The largest impact by far is by Xorg/X11 (providing the GUI) and the Nvidia setup. 

The default output for X is a monitor.  The values used for RGB is 0-255 (0 = perfect black, 255 - perfect white).  The issue comes when you use a TV as a display with source which is not created with 0-255 in mind...i.e. movies...they use 16-235 (16 = perfect black, 235 = perfect white).  Why don't they use the full range?   The shooting and digitisation process is not perfect so some head room and foot room is given, maybe to accommodate a filter overshoot for example.

So I made sure that my XBMC setup was outputting 16-235.  After a lot of research the solution was a simple addition to my xorg.conf:
Option "ColorRange""Limited"

In fact the XBMC devs have recognised this:
Video calibration...
See #Video calibration...
  Future Gotham feature: Use limited color range (16-235)  
Use limited color range instead of full color range (0-255). Limited range should be used if your display is a regular HDMI TV, while full color range should be used if your display is a PC monitor.

So now that my basic XBMC setup was done I went through the calibration steps from AVSForum guys: http://www.avsforum.com/t/948496/avs-hd-709-blu-ray-mp4-calibration. As I have the THX blue filter glasses, I was able to get the skin tones done too.

I recognise that this is not all there is to calibration and different sources will still produce different issues but this is enough for me at least.

So now I have a calibrated picture for both Sky HD and my XBMC setup...problem is when I go to other peoples homes, I am instantly distracted by the tendency of their TV's to oversaturate and use too much brightness!

Future Possibilities
I have a Canon 5D Mark III....I'm wondering whether I can use the capabilities of the camera somehow as an accurate spectrum analyser i.e. I take a picture of an RGB 128,128,128 (gray), output it on to the TV, measure with camera again, tweak until the picture is close to RGB 128,128,128....do that with multiple colours and grays etc.