Wednesday, September 16, 2015
Get list of URLs that Google Search has indexed
Here is a quick and dirty way to get the list of google results for a domain.
1. Do a google search for site:example.com
2. Open your browser's console
3. copy paste:
var elems = document.getElementsByTagName('cite')
for(var i=0; i<elems.length; i++) {
console.log(elems[i].innerText)
}
Tuesday, August 7, 2012
Mounting S3 on an Ubuntu EC2 intance
Compile S3FS
sudo apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support
curl -O http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz
tar -xzvf s3fs-1.61.tar.gz
cd s3fs-1.61/
configure
make
sudo make install
setup security
vim /etc/passwd-s3fs
accessKeyId:secretAccessKey
sudo chmod 640 /etc/passwd-s3fs
mount it
Saturday, August 4, 2012
Everything a Web Developer needs for OSX
Setting Up New Computer.
Within the last month received a new MacBook Pro, and had to re-install another. So I decided to list the software I installed.- Browsers
- Chrome www.google.com/chrome/
- Firefox www.mozilla.org/firefox
- FireBug
- LastPass lastpass.com/
- Command line Tools
- Git git-scm.com/
- php 5.4 php-osx.liip.ch/
- nmap nmap.org/
- homebrew mxcl.github.com/homebrew/
- Xcode (3.22 for SnowLeopard | 4.4 for Lion)
- IDEs
- NetBeans netbeans.org/
- OSX Java
- PhpStorm www.jetbrains.com/phpstorm/
- Sublime Text www.sublimetext.com/
- GitX(L) gitx.laullon.com/
- SourceTree www.sourcetreeapp.com/
- DataBase
- Mysql www.mysql.com/
- Sequel Pro www.sequelpro.com/
- JumpCut jumpcut.sourceforge.net/
- Other Apps
- Minuteur www.phg-home.com/index_mac.html
- ScreenFlow www.telestream.net/screen-flow/
- Tweaks
- Terminal
- Change theme to home brew
- Add command key
- When the shell exists: Close if the shell exited cleanly
- Keyboard ShortCuts Prefs
- All controls for tab key
- Shell Tweaks
- edit ~/.profile
- function parse_git_branch_and_add_brackets {
git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\ \[\1\]/'
}
PATH="/usr/local/php5/bin:/usr/local/mysql/bin:$PATH"
PS1='\[\e[01;32m\]\u@\h:\[\e[01;34m\]\w\[\e[0;31m\]$(parse_git_branch_and_add_brackets)\[\e[0m\]$ '
alias ls='ls -G'
Thursday, January 12, 2012
OpenSSL - Create your own Certificate of Authority and sign your SSL Certs
Creating your own CA for Signing Dev Certs
Here are some handy commands to create a CA and sign your own cert for development.
The commands are fully scriptable: all key passwords have been bypassed with the -nodes flag, and the interactive openssl prompts not necessary with in the the -subj flag.
tldr; Short version for the Impatient:
# Create Certificate of Authority
SUBJ="/C=US/ST=CA/L=San Diego/O=Lance Rushing/OU=Development/CN=Dev Certificate of Authority"
openssl req -subj "$SUBJ" -nodes -new -x509 -extensions v3_ca -keyout devCA.key -out devCA.crt -days 365
# Create Key and CSR
HOST="appname.local"
SUBJ="/C=US/ST=CA/L=San Diego/O=Lance Rushing/OU=Dev Web Sites/CN=$HOST"
openssl req -subj "$SUBJ" -newkey rsa:2048 -nodes -keyout $HOST.key -out $HOST.csr -days 365
# Create the Cert by signing the CSR with our CA
openssl x509 -req -CA devCA.crt -CAkey devCA.key -CAcreateserial -in $HOST.csr -out $HOST.crt
Add the appname.key and appname.crt to your apache/nginx/IIS config, and add the devCA.crt to your browser.
Long Version.
When developing webservices that will be using SSL in production, I like to use SSL in development.
The old way
The traditional way to do this is create a self-signed cert. Where you sign your Certificate Signing Request (CSR) with the same key that created it. (Thus "self-signed".)
HOST="appname.local"
SUBJ="/C=US/ST=CA/L=San Diego/O=Lance Rushing/OU=Dev/CN=$HOST"
openssl req -subj "$SUBJ" -x509 -newkey rsa:2048 -nodes -keyout $FILENAME.key -out $FILENAME.crt -days 3650 ## look mom! One line.
Then add it to the apache.conf
<VirtualHost 172.16.1.1:443>
ServerName appname.local:443
DocumentRoot "/Users/lance/Sites/AppName/src/webroot"
SSLEngine on
SSLCertificateFile "/Users/lance/Sites/AppName/certs/appname.local.crt"
SSLCertificateKeyFile "/Users/lance/Sites/AppName/certs/appname.local.key"
</VirtualHost>
This works great, but now with every new ssl cert I generate I have to add a trust exception in my browser.
A better way
To avoid multiple trust exceptions of you self-signed certs, is to first generate a "Certificate of Authority" (CA), add that CA to your browser, and then sign all of your certs with that CA.
# Create CA
SUBJ="/C=US/ST=CA/L=San Diego/O=Lance Rushing/OU=Development/CN=Dev Certificate of Authority"
openssl req -subj "$SUBJ" -new -x509 -nodes -extensions v3_ca -keyout devCA.key -out devCA.crt -days 3650
# Create Key and CSR (same as above)
HOST="appname.local"
SUBJ="/C=US/ST=CA/L=San Diego/O=Lance Rushing/OU=Dev Web Sites/CN=$HOST"
openssl req -subj "$SUBJ" -newkey rsa:2048 -nodes -keyout $HOST.key -out $HOST.csr -days 365
# Create the Cert by signing the CSR with our CA
openssl x509 -req -CA devCA.crt -CAkey devCA.key -CAcreateserial -in $HOST.csr -out $HOST.crt
# cleanup CSR
rm $HOST.csr
Thoughts
These commands are the shortest way I've found to create keys, CSRs, and signing. Other guides often use three steps (1 genrsa, 2 key export w/o pass, 3 req ) to generate the CSR, whereas it is possible to do it with 1 step.
Hint: By using the -subj flag we can bypass openssl's interactive prompts
Hint: I use Virtual IPs for each https web service I need:
OsX: $ ifconfig lo0 alias 172.16.1.1
Linux: $ ifconfig lo:0 172.16.1.1
windows: http://support.microsoft.com/kb/236869
http://gagravarr.org/writing/openssl-certs/ca.shtml
http://www.g-loaded.eu/2005/11/10/be-your-own-ca/
Wednesday, April 6, 2011
Fixing my Bash Prompt on OSX
I have been having a problem with the bash prompt. When I use the up and down arrows to scroll throw the history buffer, the output of the previous commands would get garbled and not line up. After messing around for 20 minutes I found a solution.
My old prompt:
My new prompt:
Something about changing the \033 to \e works. Why? I don't know, if you do please comment.
My old prompt:
PS1='\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[0;31m\]$(parse_git_branch_and_add_brackets)\[\033[00m\]\$ '
My new prompt:
PS1='\[\e[01;32m\]\u@\h:\[\e[01;34m\]\w\[\e[0;31m\]$(parse_git_branch_and_add_brackets)\[\e[0m\]$ '
Something about changing the \033 to \e works. Why? I don't know, if you do please comment.
Monday, March 7, 2011
BASH Prompt in OSX with git branch
I found others doing the same.
Here is the content of my ~/.profile.
Here is the content of my ~/.profile.
function parse_git_branch_and_add_brackets {
git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\ \[\1\]/'
}
PS1="\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[0;31m\]\$(parse_git_branch_and_add_brackets)\[\033[00m\]\\$ "
Saturday, February 19, 2011
Quickly rename files to lower case
Here is a simple bash oneliner to rename all folders/files in a directory to lower case
for old in * ; do new=`echo $old | tr "[[:upper:]]" "[[:lower:]"`; mv $old $new; done;
Monday, February 14, 2011
Getting PHP's pcntl working on Snow Lepard
(make sure you've installed Apple's developer tools)
1. Check your version you have installed and download the PHP source code.
2. unpack, phpize, configure, make, make install
3. Add "extension=pcntl.so" to your /etc/php.ini
4. Confirm
Done
1. Check your version you have installed and download the PHP source code.
$ php -v
PHP 5.3.3 (cli) (built: Aug 22 2010 19:41:55)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
with Xdebug v2.1.0, Copyright (c) 2002-2010, by Derick Rethans
$ curl -O http://us.php.net/distributions/php-5.3.3.tar.gz
2. unpack, phpize, configure, make, make install
$ tar -xzvf php-5.3.3.tar.gz
$ cd php-5.3.3/ext/pcntl/
$ phpize
$ ./configure
$ make
$ sudo make install
3. Add "extension=pcntl.so" to your /etc/php.ini
4. Confirm
$ php -i | grep pcntl
pcntl
pcntl support => enabled
Done
Friday, December 31, 2010
New Year's Resolutions: New Passwords for 2011
I'll admit it. I've been bad. I'm a password reuse addict. I just love the way the muscle memory makes it so easy to type. And yes, I have a couple variations on the theme, but I've known for a long time that I would have to get clean. The recent gawker compromise has given me the push I need.
A couple of years ago I've started using LastPass. LastPass offers many nifty security features, but I use it b/c I move between computers, and it fills my forms in for me. (lazy.) Well no more! Time to leverage LastPass to it's potential.
Tonight I'm going clean. No more password reuse for 2011.
A couple of years ago I've started using LastPass. LastPass offers many nifty security features, but I use it b/c I move between computers, and it fills my forms in for me. (lazy.) Well no more! Time to leverage LastPass to it's potential.
Tonight I'm going clean. No more password reuse for 2011.
Thursday, November 15, 2007
Windows Re-install
Whenever I have a fresh install of XP (twice this week) I have a set of software I need to install.
- Firefox
- Smart Bookmarks Bar
- Firebug
- Syntap's Timestamp
- Tweak UI
- UltraVNC
- Gmail Notifier
- PgAdmin III
- GIMP
- Ultraedit
- WinRar
- WinAmp
- SFTP Drive
- Pidgin
- Airset
Sunday, October 14, 2007
Monday, August 6, 2007
Recover / Reset Linux Password
Found these concise steps for reseting a password at: http://ubuntuforums.org/showthread.php?t=3609
Just thought I'd write it down for myself, instead of having google it up again.
Just thought I'd write it down for myself, instead of having google it up again.
Q: I am a complete moron and forgot my password. How can I get back into my system?
(Hey, YOU forgot it, I think that gives me the right to do some degrading before letting you back in!)
A:
1. Turn off your computer.
2. Tap it 3 times
3. Say your favorite magic word.
---- If that doesn't fix your problem, do the steps below... #-o
4. Turn your computer on.
5. Press ESC at the grub prompt.
6. Press e for edit.
7. Highlight the line that begins kernel ........., press e
8. Go to the very end of the line, add rw init=/bin/bash
9. press enter, then press b to boot your system.
10. Your system will boot up to a passwordless root shell.
CAUTION: This is a FULL ROOT SHELL! You can damage your system if not careful!
11. Type in passwd. Set your password.
12. Type in reboot.
NOTE: steps 1-3 may help at this point.
13. Bow down to me....
Tuesday, June 26, 2007
Initial Postgres Setup for Ubuntu
After installing postgresql, here are the first few things I usually do. I'm usually running Ubuntu, but these would be necessary for any postgres install.
- Create database users
$ sudo su - postgres # become the postgres user
$ createuser -P lance # I'm usually a super user
$ exit # exit out of the postgres user
$ createuser account1 ... # Create the standard accounts - Setup conf files
$ sudo vi /etc/postgresql/8.2/main/pg_hba.conf
$ sudo vi /etc/postgresql/8.2/main/postgresql.conf - Move Data to separate partition (/data)
$ sudo /etc/init.d/postgresql-8.2 stop
$ sudo mv /var/lib/postgresql/8.2 /data/postgresql/8.2
$ sudo ln -fs /data/postgresql/8.2/main /etc/postgresql/8.2/main/pgdata
$ sudo /etc/init.d/postgresql-8.2 start - Setup admin pack
$ sudo -u postgres psql < /usr/share/postgresql/8.2/contrib/adminpack.sql
plr for postgres 8.2 on Ubuntu Feisty using Gutsy deb-src
The only reason I have not upgraded to postgresql 8.2 is lack of a plr package.
After yesterday's dissection of the plr install for postgres 8.1, I decided to create a 8.2 install.
I noticed plr 8.2 is already available for Gutsy (which is still in development at the time of this writing).
http://packages.ubuntu.com/gutsy/libs/postgresql-8.2-plr
So I tried the Gutsy .deb (binary), but it would not install b/c of a unmet gcc dependency. So I tried building it from the deb sources, but plr.c would not compile as is. After hacking around I was able to get it to build.
Using Gutsy deb-src to build a Feisty .deb (binary) for postgesql 8.2 plr
Remove the ", R_NilValue" from the end of the line. Line should read:
#define R_PARSEVECTOR(a_, b_, c_) R_ParseVector(a_, b_, (ParseStatus *) c_)
After yesterday's dissection of the plr install for postgres 8.1, I decided to create a 8.2 install.
I noticed plr 8.2 is already available for Gutsy (which is still in development at the time of this writing).
http://packages.ubuntu.com/gutsy/libs/postgresql-8.2-plr
So I tried the Gutsy .deb (binary), but it would not install b/c of a unmet gcc dependency. So I tried building it from the deb sources, but plr.c would not compile as is. After hacking around I was able to get it to build.
Using Gutsy deb-src to build a Feisty .deb (binary) for postgesql 8.2 plr
$ wget http://mirrors.easynews.com/linux/ubuntu/pool/universe/p/plr/plr_8.2.0.1-1.diff.gz
$ wget http://mirrors.easynews.com/linux/ubuntu/pool/universe/p/plr/plr_8.2.0.1-1.dsc
$ wget http://mirrors.easynews.com/linux/ubuntu/pool/universe/p/plr/plr_8.2.0.1.orig.tar.gz
$ dpkg-source -x plr_8.2.0.1-1.dsc
$ cd plr-8.2.0.1
$ vi +185 plr.h
Remove the ", R_NilValue" from the end of the line. Line should read:
#define R_PARSEVECTOR(a_, b_, c_) R_ParseVector(a_, b_, (ParseStatus *) c_)
$ dpkg-buildpackage -rfakeroot -b
$ cd ..
$ sudo dpkg --install postgresql-8.2-plr_8.2.0.1-1_amd64.deb
Monday, June 25, 2007
Ubuntu: postgresql-8.1-plr for Feisty missing plr.so and plr.sql
I am trying to get plr installed. However after running apt-get install postgresql-8.1-plr I get the dreaded:
I double check the package list for plr
I check around on the ubuntu packages and confirm that other releases have the the .so file:
http://packages.ubuntu.com/dapper/libs/postgresql-8.1-plr
http://packages.ubuntu.com/edgy/libs/postgresql-8.1-plr
So I file a bug and start looking into a way to fix it.
I search for how to build a package: and come up the official guide.
The resulting postgresql-8.1-plr_0.6.2-4ubuntu1_amd64.deb still is missing the files.
After a lot of poking at the problem I notice that the binary files are created in a directory called install-8.1. However in that dir, are directories for 8.2.
After tracking the problem down, I created a diff file:
And then I uploaded to launchpad.
Download the patch file to your home directory as rules.patch, then do:
ERROR: could not access file "$libdir/plr": No such file or directoryI double check the package list for plr
$ dpkg-query --listfiles postgresql-8.1-plr
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/postgresql-8.1-plr
/usr/share/doc/postgresql-8.1-plr/index.html
/usr/share/doc/postgresql-8.1-plr/plr-aggregate-funcs.html
/usr/share/doc/postgresql-8.1-plr/plr-data.html
/usr/share/doc/postgresql-8.1-plr/plr-func-naming.html
/usr/share/doc/postgresql-8.1-plr/plr-funcs.html
/usr/share/doc/postgresql-8.1-plr/plr-global-data.html
/usr/share/doc/postgresql-8.1-plr/plr-install.html
/usr/share/doc/postgresql-8.1-plr/plr-license.html
/usr/share/doc/postgresql-8.1-plr/plr-module-funcs.html
/usr/share/doc/postgresql-8.1-plr/plr-overview.html
/usr/share/doc/postgresql-8.1-plr/plr-pgsql-support-funcs.html
/usr/share/doc/postgresql-8.1-plr/plr-spi-rsupport-funcs.html
/usr/share/doc/postgresql-8.1-plr/plr-trigger-func.html
/usr/share/doc/postgresql-8.1-plr/stylesheet.css
/usr/share/doc/postgresql-8.1-plr/copyright
/usr/share/doc/postgresql-8.1-plr/changelog.Debian.gzHmm.. where's the .so???I check around on the ubuntu packages and confirm that other releases have the the .so file:
http://
http://
So I file a bug and start looking into a way to fix it.
I search for how to build a package: and come up the official guide.
$ mkdir -p ~src/plr/feisty
$ cd ~src/plr/feisty
$ apt-get source postgresql-8.1-plr
$ sudo apt-get build-dep postgresql-8.1-plr
$ cd plr-0.6.2
$ dpkg-buildpackage -rfakeroot -uc -b
$ cd ..
The resulting postgresql-8.1-plr_0.6.2-4ubuntu1_amd64.deb still is missing the files.
$ dpkg --contents postgresql-8.1-plr_0.6.2-4ubuntu1_amd64.debAfter a lot of poking at the problem I notice that the binary files are created in a directory called install-8.1. However in that dir, are directories for 8.2.
After tracking the problem down, I created a diff file:
diff -u debian/rules /home/lance/rules.patched > /home/lance/rules.patchAnd then I uploaded to launchpad.
--- debian/rules 2007-06-25 15:37:43.000000000 -0700
+++ /home/lance/rules.patched 2007-06-25 15:37:23.707021486 -0700
@@ -47,7 +47,7 @@
dh_testroot
# 8.1
- $(MAKE) -C build-8.1/plr -f Makefile.pgxs R_HOME=/usr/lib/R install DESTDIR=$(HERE)/install-8.1 PGXS=/usr/lib/postgresql/8.1/lib/pgxs/src/makefiles/pgxs.mk
+ $(MAKE) -C build-8.1/plr -f Makefile.pgxs R_HOME=/usr/lib/R install DESTDIR=$(HERE)/install-8.1 PGXS=/usr/lib/postgresql/8.1/lib/pgxs/src/makefiles/pgxs.mk datadir=/usr/share/postgresql/8.1 docdir=/usr/share/doc/postgresql-doc-8.1 pkglibdir=/usr/lib/postgresql/8.1/lib
# install files
dh_installdocs -a
Download the patch file to your home directory as rules.patch, then do:
$ mkdir /tmp/plr
$ cd /tmp/plr
$ apt-get source postgresql-8.1-plr
$ sudo apt-get build-dep postgresql-8.1-plr
$ cd plr-0.6.2
$ patch -p0 -i ~/rules.patch
$ dpkg-buildpackage -rfakeroot -uc -b
$ cd ..
$ sudo dpkg --install postgresql-8.1-plr_0.6.2-4ubuntu1_amd64.deb
Transfering multiple files with netcat
Today I needed to backup some files from one server to another. The servers sit next to one another on the network. Usually I use scp or rsync to move files around, but today I felt like trying something different.
First I considered using FTP. However, I do not like running FTP servers and didn't feel like configuring the daemon.
Then I thought of netcat. A quick Google search got me this page: http://www.oreillynet.com/pub/h/1058
However I want to transfer multiple files (all in the same directory). So I decided to use tar, and instead of creating a file, I would pipe it's output through netcat.
On the Receiving Computer (Server):
And on the Sending Computer (Client):
The files transfered fast (2 min 20 seconds for 1.6 Gigs). I ran a md5sum on the client and server to confirm, and everything was perfect.
First I considered using FTP. However, I do not like running FTP servers and didn't feel like configuring the daemon.
Then I thought of netcat. A quick Google search got me this page: http://www.oreillynet.com/pub/h/1058
However I want to transfer multiple files (all in the same directory). So I decided to use tar, and instead of creating a file, I would pipe it's output through netcat.
On the Receiving Computer (Server):
nc -v -w 30 -p 5600 -l | tar -x
And on the Sending Computer (Client):
tar -c * | netcat -v -w 2 10.0.0.2 5600;
The files transfered fast (2 min 20 seconds for 1.6 Gigs). I ran a md5sum on the client and server to confirm, and everything was perfect.
Sunday, June 24, 2007
Guides and Tools
List of Guides and tools I've found on the Net.
(This is like a bookmarks.)
Web Site Administration:
Breaking Stuff:
(This is like a bookmarks.)
Web Site Administration:
- Pingdom's full page test looks very interesting:
http://tools.pingdom.com/fpt/
Breaking Stuff:
- A little guide/intro to aircrack
http://www.grape-info.com/doc/win2000srv/security/aircrack-2.3.html - Windows Assembly Tutorials. Great for taking stuff apart.
http://win32assembly.online.fr/tutorials.html
First Post
Another blog. After years of poking at blogging software (Drupal, different Nukes, MT, GeekLog, etc), I've decided not to invest anymore time is investigating software. Instead, my limited blogging time, I should invest in content.
And while I've always know content to be king, I've never practiced what I preach. Instead I usually get caught up in the solving the puzzle of installing and running different pieces of software. And while that is a worth while endeavor, I no longer have the extra amount for those pursuits.
I've started using Google's services for some of my mail hosting, and have been pleased. So, it's about time I tried out blogger.com/Google's blogging software.
So here I come Brave new world of blogging. This blog is primarily a tool for myself, but I hope anything I contribute will be useful for others.
And while I've always know content to be king, I've never practiced what I preach. Instead I usually get caught up in the solving the puzzle of installing and running different pieces of software. And while that is a worth while endeavor, I no longer have the extra amount for those pursuits.
I've started using Google's services for some of my mail hosting, and have been pleased. So, it's about time I tried out blogger.com/Google's blogging software.
So here I come Brave new world of blogging. This blog is primarily a tool for myself, but I hope anything I contribute will be useful for others.
Subscribe to:
Posts (Atom)